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SUMMARY  OF  THE  STUDY 

The  Chief  of  Naval  Education  and  Training  (CNET)  tasked  the  Training 
Analysis  and  Evaluation  Group  (TAEG)  to  assess  the  feasibility  and  desirabil¬ 
ity  of  obtaining  training  feedback  from  petty  officers  who  rotate  from  fleet 
billets  to  the  Naval  Education  and  Training  Command  (NAVEDTRACOM)  to  attend 
instructor  training  (IT)  or  "(^-level  courses  in  their  rating.  Development 
and  assessment  of  a  technique  for  collecting  feedback  within  this  context 
were  included  in  the  tasking.  The  project  involved  collecting  specific 
feedback  about  training  given  in  the  six  "A"-level  courses  which  serve  the 
ratings  listed  below: 

•  Aviation  Machinist's  Mate  (AD) 

•  Machinery  Repairman  (MR) 

•  Engineman  (EN) 

•  Mess  Management  Specialist  (MS) 

•  Aviation  Electronics  Technician  (AT) 

•  Aviation  Fire  Control  Technician  ( AQ) 

•  Aviation  Antisubmarine  Warfare  Technician  (AX) 

•  Fire  Control  Technician  (FT). 

A  structured  interview  procedure  was  used  for  data  collection.  Two 
hundred  and  eighty-one  petty  officers  attending  IT  and  "C'-level  courses 
were  interviewed  at  their  schools.  In  addition,  82  MS  petty  officers  occupying 
fleet  billets  provided  feedback  about  MS  "A"  school  training.  Interviews 
were  conducted  by  civilian  and  military  personnel  regularly  assigned  to  the 
participating  activities.  Interviewer  training  was  accomplished  by  members  of 
the  TAEG  staff. 

The  general  conclusions  of  this  study  program  are: 

•  It  is  both  feasible  and  desirable  to  collect  training  feedback 
information  from  petty  officers  recently  transferred  from  fleet 
billets  to  attend  advanced  schools  within  the  NAVEDTRACOM. 

•  The  structured  interview  procedure  yields  useful  and  valuable 
information  for  curriculum  evaluation. 

These  conclusions  are  supported  by  the  following  specific  findings: 

1.  A  review  of  the  background  characteristics  of  petty  officers 
attending  advanced  schools  within  the  NAVEDTRACOM  and  the  recency  of  their 
fleet  assignments  led  to  the  conclusion  that  feedback  provided  by  them  would 
not  differ  substantially  from  feedback  that  could  be  gathered  from  their 
counterparts  still  serving  in  operational  fleet  billets.  Thus,  training 
feedback  from  the  school  petty  officer  groups  who  were  interviewed  during 
the  study  program  was  considered  to  be  valid. 

2.  The  structured  interview  method  yields  valuable  data  for  curriculum 
review.  The  data  are  useful  for  identifying  training  deficiencies  and  the 
nature  of  those  deficiencies. 
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3.  Approximately  60  percent  of  the  petty  officers  assigned  to  advanced 
courses  within  the  NAVEDTRACOM  were  qualified  to  evaluate  the  fleet  job  per¬ 
formance  of  recent  "A"  school  graduates. 

4.  The  Avionics  ( AV ) ^  portion  of  the  study  program  demonstrated  that  essentially 
the  same  information  about  "A"  school  graduate  fleet  job  performance  can  be 

obtained  from  either  IT  school  students  or  more  junior  "C"  school  students. 

This  is  particularly  important  since  "C"  school  students  comprise  the  majority 
of  advanced  school  attendees. 

5.  The  MS  portion  of  the  study  demonstrated  statistically  that  feedback 
obtained  from  personnel  attending  advanced  schools  within  the  NAVEDTRACOM 

was  equivalent  to  feedback  obtained  directly  from  fleet  sources.  This 
finding  reinforces  the  assumption  made  above.  The  MS  effort  also  demonstrated 
that  the  structured  interview  method  can  be  used  successfully  to  collect 
training  feedback  within  the  fleet. 

6.  No  "complaints"  were  voiced  concerning  the  use  of  school  facilities, 
staff  for  technical  assistance  and  interviewing,  or  student  time.  No  reasons 
became  evident  to  assume  that  use  of  the  method  within  the  schools  was  unde¬ 
sirable  from  an  "inconvenience/disruption"  of  routine  standpoint. 

7.  The  average  time  required  to  complete  individual  interviews  at  all 
six  schools  was  approximately  1  hour.  This  met  the  expressed  desires  of 
school  commands. 

8.  Interviewing  duties  can  be  shared  among  a  variety  of  school 
personnel  inexperienced  in  interviewing  techniques.  As  long  as  the  proce¬ 
dures  are  followed  in  a  reasonable  way,  useful  data  can  be  obtained.  However, 
better  training  for  interviewers  would  undoubtedly  have  improved  the  quality 
of  comments  describing  the  specific  nature  of  training  problems. 

In  view  of  the  findings  and  school  needs  for  a  continuing  flow  of  feed¬ 
back  information,  it  is  recommended  that  the  structured  interview  method 
assessed  during  the  study  program  be  used  on  a  routine  basis  to  collect 
training  feedback  within  the  advanced  school  context.  The  appendix  to  this 
report  provides  detailed  procedures  for  implementing  and  conducting  feedback 
data  collection  programs  using  the  method. 
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SECTION  I 
INTRODUCTION 

The  Training  Analysis  and  Evaluation  Group  (TAEG)  was  tasked  by  the 
Chief  of  Naval  Education  and  Training  (CNET)  to  examine  the  feasibility  and 
desirability  of  obtaining  training  feedback  information  from  petty  officers 
recently  transferred  from  the  fleet  to  attend  advanced  courses  within  the 
Naval  Education  and  Training  Consnand  (NAVEDTRACOM).  The  tasking  included  a 
requirement  to  develop  and  evaluate  a  method  suitable  for  the  systematic 
collection  of  feedback  information  within  NAVEDTRACOM  schools.  The  tasking 
further  stipulated  that  necessary  work  be  performed  with  at  least  six  Navy 
ratings  and  that  all  feedback  obtained  be  provided  to  participating  schools 
for  use  in  curriculum  evaluation. 

BACKGROUND 

To  acquire  the  information  and  experience  necessary  to  accomplish  the 
tasking,  six  "A"-level  courses/schools  were  selected  for  evaluation.  These 
serve  the  eight  ratings  listed  below: 

•  Aviation  Machinist's  Mate  (AD) 

•  Machinery  Repairman  (MR) 

•  Engineman  (EN) 

•  Mess  Management  Specialist  (MS) 

•  Aviation  Electronics  Technician  (AT) 

•  Aviation  Fire  Control  Technician  (AQ) 

•  Aviatien  Antisubmarine  Warfare  Technician  (AX) 

•  Fire  Control  Technician  (FT). 

A  structured  interview  method  was  used  to  obtain  training  feedback  data 
from  petty  officers  attending  Instructor  Training  (IT)  or  "C"-level  courses 
in  their  ratings.  The  data  were  used  to  assess  each  "A"  school's  curriculum 
in  terms  of  the: 

•  relevancy  of  a  school's  training  for  graduates'  fleet  job 
assignments 

•  graduates'  fleet  performance  of  job  tasks  for  which  they  received 
school  training. 

The  evaluation  data  obtained  during  the  program  were  provided  to  the 
schools  in  six  previous  TAEG  reports.  These  are  listed  below  for  the 
ratings  involved: 

•  AD  -  Technical  Memorandum  79-3  (ref.  12) 

•  MR  -  Technical  Memorandum  79-4  (ref.  2) 

•  EN  -  Technical  Memorandum  79-5  (ref.  11) 

•  MS  -  Technical  Report  No.  76  (ref.  6) 

•  AT,  AQ,  AX  -  Technical  Memorandum  80-4  (ref.  7) 

•  FT  -  Technical  Memorandum  80-5  (ref.  10) 
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PURPOSE 

This  study  assessed  the  feasibility  and  desirability  of  obtaining 
training  feedback  information  from  petty  officers  attending  advanced  schools 
within  the  NAVEDTRACOM. 

The  report  summarizes  the  total  effort  conducted  in  response  to  the  CNET 
tasking.  It  is  the  final  report  of  a  work  program  initiated  in  December  1977. 
It  uses  data,  information,  and  experience  gained  within  the  six  NAVEDTRACOM 
school  contexts  to  provide: 

•  an  assessment  of  the  feasibility  and  desirability  of  collecting 
training  feedback  information  from  advanced  students  within  the 
NAVEDTRACOM 

•  evaluative  information  concerning  the  method  developed  for  feedback 
data  collection. 

In  addition,  the  report  provides  recommendations  and  procedures  for 
future  use  of  the  method  for  feedback  data  collection. 

ORGANIZATION  OF  THE  REPORT 

The  remainder  of  this  report  is  contained  in  three  sections  and  an 
appendix.  Section  II  presents  the  technical  approach.  It  provides  descrip¬ 
tions  of  the  instruments  and  procedures  used  for  data  collection.  It  also 
describes  techniques  used  for  data  reduction  and  analysis  and  for  evaluating 
the  data  collection  method.  The  major  findings  of  the  program  are  given 
in  section  III.  These  are  discussed  both  in  relation  to  the  program  objec¬ 
tives  and  to  future  use  of  the  structured  interview  method.  Section  IV 
presents  conclusions  and  recommendations.  The  appendix  contains  guidelines 
for  use  of  the  method  in  future  training  appraisal  efforts.  Guidance  is 
presented  for  instrument  development  and  use  and  data  reduction  and  inter¬ 
pretation.  These  procedures  are  presented  as  an  aid  to  the  schools  for 
implementing  and  conducting  training  appraisal  efforts  with  locally  available 
resources. 
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SECTION  II 
TECHNICAL  APPROACH 

This  section  presents  the  technical  approach  used  in  this  study  program. 

A  clarification  of  the  issues  involved  is  presented  first.  This  is  followed 
by  information  describing  study  planning,  procedures/criteria  used  to 
select  schools,  and  factors  influencing  the  choice/design  of  the  data  collec¬ 
tion  method.  Descriptions  of  the  data  collection  instruments  and  procedures 
and  techniques  used  for  data  reduction  and  analysis  are  also  presented.  In 
addition,  information  is  given  describing  procedures  used  to  evaluate  the 
structured  interview  method. 

CLARIFICATION  OF  ISSUES 

A  definitive  assessment  of  the  feasibility  and  desirability  of  obtaining 
training  feedback  information  from  petty  officers  attending  advanced  NAVEDTRACOM 
schools  required  experience  and  data  collection  within  the  school  environment. 
The  feasibility  question  involved  concerns  such  as  the  availability  of 
"qualified"  petty  officers  from  whom  feedback  could  be  obtained  and  the 
ability  of  the  schools  to  conduct/support  a  data  collection  effort.  Assess¬ 
ment  of  the  desirability  of  obtaining  feedback  data  within  the  school  environ¬ 
ment  required  consideration  for  data  validity  and  the  value/usefulness  of 
the  data  for  curriculum  review  purposes.  Information  from  both  areas; 
i.e.,  "feasibility"  and  "desirability,"  was  relevant  to  the  utility  and 
value  of  the  method  for  data  collection  and  to  recommendations  concerning 
its  future  use  in  feedback  data  collection.  The  program  was  organized  to 
obtain  the  necessary  information. 

STUDY  PLANNING 

To  assist  subsequent  decision  making  about  the  conduct  of  the  study, 
discussions  were  held  early  in  the  program  with  staff  personnel  at  various 
NAVEDTRACOM  activities.  These  included  the  staffs  of  the  Chief  of  Naval 
Technical  Training  (CNTECHTRA);  Naval  Air  Technical  Training  Center  (NATTC), 
Memphis;  Service  School  Command  (SERVSCOLCOM) ,  Great  Lakes  and  San  Diego; 
Commander  Training  Command,  U.S.  Atlantic  Fleet  (COMTRALANT) ;  Commander, 

Training  Command,  U.S.  Pacific  Fleet  (COMTRAPAC);  and  Fleet  Training  Center 
(FLETRACEN),  Norfolk.  The  discussions  centered  on: 

•  selection  of  "A"  schools  for  "evaluation" 

•  training  evaluation  needs  and  philosophies 

•  selection/development  of  feedback  data  collection  methods  and 
procedures  suitable  for  use  in  the  school  environment. 

SCHOOL  SELECTION.  The  CNET  tasking  stipulated  that  work  be  conducted  with 
a  minimum  of  six  ratings  (unspecified).  The  six  "A"  schools  in  the  study 
were  selected  by  TAEG  and  CNET  015  staff  following  conversations  with  and 
recommendations  made  by  appropriate  command  personnel.  A  number  of  criteria 
were  applied  to  the  selection  of  schools.  These  included: 

•  identification  by  appropriate,  responsible,  local  command  personnel 
of  schools/courses  which  were  in  need  of  evaluative  feedback 
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•  unclassified  curricula 

•  geographic  representation  of  advanced  schools,  especially  IT 

schools 

•  representative  sample  of  Navy  jobs 

•  sufficient  numbers  of  personnel  in  the  ratings  of  interest  projected 

for  assignment  to  the  advanced  schools  to  permit  a  reliable 
assessment  of  the  course  and  the  data  collection  method  within  a 
reasonable  period  of  time 

•  willingness  of  the  "A"  school  to  support  the  data  collection 
effort  (e.g.,  provide  subject  matter  expertise  for  instrument 
development,  personnel  to  conduct  interviews,  adequate  facilities). 

The  first  six  schools  which  met  these  principal  criteria  were  selected 
for  evaluation. 

METHOD  DESIGN  CONSIDERATIONS.  During  the  early  program  discussions,  school 
personnel  cited  needs  for  feedback  information  which  could  be  used  to 
identify  training  deficiencies  and  at  the  same  time  provide  specific  detail 
concerning  the  nature  of  any  such  deficiencies.  They  also  expressed  the 
view  that,  to  minimize  staff  and  student  involvement,  data  collection  from 
any  one  individual  should  not  exceed  approximately  1  hour. 

Subsequently,  the  decision  was  made  to  develop  a  structured  interview 
method  for  data  collection  at  the  schools.  It  was  believed  that  this  method 
could  yield  more  detailed  information  than  other  possible  methods  (see  ref. 

9).  A  number  of  other  considerations  also  supported  this  decision.  Previous 
TAEG  studies  had  already  produced  a  viable  questionnaire  method  for  collect¬ 
ing  feedback  data  (see  refs.  3  and  4).  CNET  desired  that  an  additional 
method  be  developed  and  evaluated.  A  structured  interview  method  could  be 
used  more  readily  in  the  school  context  than  in  the  fleet  setting  because 
of  the  easier  CNET  access  to  potential  interviewees.  It  was  also  assumed 
that,  overall,  there  would  be  fewer  competing  demands  on  the  time  of  petty 
officers  attending  school  than  on  those  in  operational  fleet  billets. 

In  designing  instruments  and  procedures  to  conduct  interviews,  full 
consideration  was  given  to  the  realities  of  the  school  context  and  the 
desires  of  school  staff.  Accordingly,  design  features  were  deliberately 
incorporated  to  facilitate  the  schools'  implementation  of  the  method,  tr> 
minimize  time  requirements  for  data  collection,  and  to  obtain  detailed 
information  about  possible  training  deficiencies. 

STRUCTURED  INTERVIEW  METHOD 

The  structured  interview  concepts  employed  were  adapted  from  procedures 
previously  used  in  fleet  feedback  projects  (for  example,  refs.  1  and  13). 

The  interview  was  designed  to  acquire  specific  data  concerning  a  recent 
graduate's  performance  of  job  tasks  in  the  fleet  for  which  he  had  received 
training  at  his  "A"  school.  Three  categories  of  job  task  information  were 
of  interest.  These  involved  school-trained  tasks  that  recent  graduates: 
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•  did  not  perform  in  their  fleet  assignments 

•  performed  on  the  job  but  with  difficulty 

•  performed  on  the  job  without  difficulty. 

The  first  category  provides  information  for  assessing  the  relevancy  of 
school  training  to  operational  job  requirements.  The  second  identifies 
performance  difficulties  which  may  be  correctable  by  school  training. 

More  specific  information  regarding  reasons  either  for  nonperformance  or 
for  performance  difficulties  was  solicited  during  the  interview.  The  third 
category  identifies  tasks  for  which  remedial  training  need  not  be  considered. 
However,  it  can  provide  information  concerning  possible  overtraining  by  a 
school. 

DESCRIPTION  OF  INTERVIEW  INSTRUMENTS.  Three  forms  were  developed  for  data 
collection: 

•  a  Background  Data  Form  for  collecting  information  about  the 
school  attendees  and  selecting  individuals  to  be  interviewed 

•  a  Feedback  Data  Form  to  guide  the  interview  and  for  recording 
data 

•  a  Reason  Code  Sheet  which  listed 

..  possible  reasons  for  graduate  difficulty  in  task  performance 
..  reasons  why  graduates  did  not  perform  some  tasks. 

These  forms  are  described  further  below. 

Background  Data  Forms.  Background  Data  Forms  were  used  with  each  rating  to 
obtain  information  about  the  individuals  attending  IT  and  "C"  schools 
within  the  NAVEDTRACOM.  Data  requested  included:  rate.  Navy  Enlisted 
Classifications  (NEC),  billet  titles,  type  and  location  of  current  and 
previous  duty  stations,  and  length  of  service.  This  type  of  information 
was  desired  to  permit  sorting  and  comparing  of  interview  data  on  the 
basis  of  selected  background  variables.  Questions  on  the  forms  also 
addressed  opportunity  to  observe  recent  "A"  school  graduates  perform  in 
their  fleet  jobs,  number  of  graduates  observed,  and  length  of  observation. 
This  information  was  needed  to  determine  whom  among  the  advanced  school 
students  could  be  considered  qualified  evaluators  of  graduate  job  per¬ 
formance.  Arbitrarily,  a  "qualified  evaluator"  was  defined  as  an  individual 
who  within  the  past  year  had  observed  the  fleet  performance  of  at  least 
one  recent  "A"  school  graduate  for  a  minimum  of  3  months.  Only  individuals 
meeting  this  criterion  were  interviewed.  In  addition  to  the  purposes 
noted  above,  background  data  were  also  used  to  describe  and  summarize  the 
characteristics  of  the  groups  from  whom  feedback  data  were  obtained  and 
to  assess  the  representativeness  of  school  groups  to  appropriate  fleet 
groups.  This  issue  bears  on  the  question  of  validity  of  data  from  the 
school  source.  A  sample  Background  Data  Form  is  contained  in  the  appendix. 
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Feedback  Data  Form.  The  Feedback  Data  Form  was  the  primary  instrument 
used  for  the  structured  interviews.  The  form  was  divided  into  two  parts. 

The  first  part,  "A,"  could  be  completed  by  respondents  prior  to  the 
interview.  The  second  part,  "B,"  was  designed  for  the  interviewer's  use 
in  conducting  the  interview  and  for  recording  comments. 

The  first  page  of  the  Feedback  Data  Form  contained  instructions  for 
completion  of  part  A.  The  left  hand  column  listed  specific  job  tasks  for 
a  given  rating.  Subsequent  columns  in  part  A  provided  space  for  respondents 
to  select  one  of  the  three  alternatives/categories  for  each  task.  Category 
selection  was  based  on  interviewee  observations  of  a  typical  "A"  school 
graduate  during  the  graduate's  first  6  months  of  duty  in  the  unit.  If  the 
specific  tasjc  was  not  usually  done  by  the  typical,  recent  graduate,  the 
"Don't  Do"  category  was  to  be  checked.  If  the  task  was  done  and  the  graduate 
had  no  difficulty  in  performing  it,  the  "Do  With  Ease"  category  was  checked. 

If  the  task  was  done  but  the  graduate  had  difficulty  performing  it,  the  "Do 
With  Difficulty"  category  was  checked.  At  all  schools,  the  option  of  having 
the  respondent  complete  part  A  prior  to  being  interviewed  was  selected.  Time 
to  complete  part  A  was  recorded.  A  sample  Feedback  Data  Form  is  contained  in 
the  appendix. 

A  standard  procedure  was  used  across  all  ratings/schools  for  initial 
development  of  the  job  task  statements  used  on  the  Feedback  Data  Forms.  In 
each  instance,  TAEG  compiled  a  task  list  for  a  rating  from  the  current  Naval 
Occupational  Task  Analysis  Program  (NOTAP)  job  task  inventory  for  that  rating. 
All  technical  job  tasks  performed  by  20  percent  or  more  of  E-3s  were  included 
in  these  initial  listings.  The  task  lists  were  reviewed  by  the  appropriate 
school  staff  for  clarity,  specificity,  and  relevance  to  their  "A"  school 
curriculum.  The  listings  were  revised  by  school  staff  to  include  only  those 
tasks  for  which  the  particular  "A"  school  actually  provided  training.  In 
some  cases,  tasks  not  included  on  an  initial  NOTAP-derived  listing  were  added 
by  school  staff  as  items  for  which  feedback  was  especially  desired. 

The  FT  school  considered  the  NOTAP  job  task  inventory  to  be  unsuitable 
for  the  development  of  task  statements  for  that  school's  Feedback  Data  Form. 
The  FT  "A"  school  curriculum  is  given  in  two  phases.  Individuals  with  a  4 
year  service  obligation  (4  YOs)  receive  only  Phase  I  of  the  curriculum. 

Those  with  a  6  year  obligation  (6  YOs)  receive  both  phases  plus  "C"  school 
training  relevant  to  anticipated  future  assignments.  The  FT  school  staff 
stated  that  the  NOTAP  list  was  more  relevant  to  the  fleet  work  of  6  YOs  than 
to  4  YOs.  The  4  YOs,  consonant  with  their  training,  are  normally  expected  to 
perform  only  the  more  basic  subtasks  related  to  a  listed  NOTAP  task.  Since 
the  FT  portion  of  the  study  was  concerned  only  with  the  phase  I  curriculum, 
the  task  statements  developed  for  the  FT  Feedback  Data  Form  reflected  only 
these  basic  subtasks. 

Part  B  of  the  Feedback  Data  Form  was  completed  by  the  interviewer 
during  the  interview  session.  The  interviewer  reviewed  the  respondent's 
selection  of  task  categories  (part  A)  and  solicited  and  recorded  reasons  why 
tasks  were  classified  as  "Don't  Do"  or  "Do  With  Difficulty."  A  list  of 
suggested  reasons,  "Reason  Codes,"  was  provided  the  respondent  for  non¬ 
performance  of  a  task  or  for  task  performance  difficulty.  The  inter¬ 
viewer  recorded  the  interviewee's  choices  of  reason  codes  and  all  other 
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amplifying  information/comments  obtained.  Finally,  the  interviewer  asked 
if  the  respondent  had  any  additional  comments  to  make  concerning  improvement 
of  "A"  school  training.  These  were  recorded  on  the  last  sheet  of  the  Feed¬ 
back  Data  Form.  At  the  completion  of  the  interview,  the  interviewer  recorded 
the  time  required  on  the  cover  sheet. 

Reason  Codes.  Reason  codes  consisting  of  letters  representing  possible 
reasons  for  nonperformance  or  for  difficulty  of  performance  were  developed 
for  use  during  interviews.  The  letter  codes  provided  a  shorthand  method 
of  recording  data,  served  as  examples  of  the  kind  and  level  of  explanatory 
information  sought,  simplified  manual  data  recording,  and  permitted 
machine  processing. 

For  each  rating/school,  a  list  of  "reasons"  why  tasks  were  not 
performed  or  were  performed  with  difficulty  was  initially  devised  by  TAEG. 
Each  list  was  reviewed  by  the  appropriate  school  staff  to  insure  its 
applicability  to  the  rating.  An  expanded  list  of  "reasons"  is  contained 
in  the  appendix.  Some  examples  are  presented  below. 

Examples  of  reasons  used  with  "Don't  Do"  selections  are: 

•  Task  not  done  on  my  ship/station/aircraft/system. 

•  Task  not  expected  of  someone  in  recent  graduate's  rate  or  level 
of  experience. 

•  Task  expected  of  recent  graduates  but  average  graduate  unable 
to  perform. 

Examples  of  reasons  used  for  "Do  With  Difficulty"  selections  are: 

•  Doesn't  know  which  tools  or  equipment  to  use. 

•  Doesn't  know  how  to  use  tools  or  equipment  properly. 

•  Lack  of  proper  equipment  to  accomplish  task. 

•  Doesn't  know  how  to  use  technical  manuals/publications  or  other 
written  references  properly. 

During  the  interview  sessions,  interviewers  solicited  amplifying 
information  supporting  the  interviewee's  selection  of  specific  reason 
codes  and  concerning  which  aspects  of  the  job  tasks  were  difficult  for 
graduates  to  perform. 

INTERVIEWEES.  Within  the  six  NAVEDTRACOM  schools,  281  individuals  were 
interviewed.  These  were  petty  officers  who: 

•  had  recently  returned  from  the  fleet  to  attend  IT  school  or 
"C"-level  courses 

•  held  one  of  the  eight  ratings  of  interest  to  the  program 
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•  had  observed  the  fleet  job  performance  of  recent  graduates  of 
the  appropriate  "A"  school  for  a  minimum  period  of  3  months 
within  the  immediately  preceding  year. 

For  the  MS  portion  of  the  program,  82  petty  officers  still  serving  in 
operational  fleet  billets  (who  met  the  third  qualification  listed  above) 
were  interviewed  to  obtain  informaMon  about  the  fleet  job  performance  of 
recent  MS  "A"  school  qraduates.  Data  from  this  fleet  group  were  used  to 
assess  directly  the  validity  of  data  obtained  from  MS  petty  officers 
within  the  NAVEDTRACOM.  This  topic  is  discussed  more  fully  below. 

To  provide  sufticient  data  on  which  to  base  reliable  conclusions,  a 
decision  was  made  to  continue  data  collection  at  each  activity  until  a 
minimum  of  30  interviews  had  been  completed.  Because  of  infrequent 
scheduling  of  petty  officers  to  the  IT  and  their  respective  "C"  schools, 
and  the  fact  that  not  all  attending  had  observed  "A"  school  graduate  per¬ 
formance,  data  collection  required  longer  periods  of  time  than  originally 
anticipated.  (See  section  III.) 

OAT A  COLLECTION.  Our  inq  initial  project  coordination  visits  to  the  schools, 
regularly  assigned  civilian  education  and  training  specialists  and/or 
military  staff  were  identified  by  the  cognizant  commands  to  conduct  inter¬ 
views  of  the  advances  students.  A:  all  schools,  the  local  Curriculum 
Instructional  Standards  Office  (CISO)  monitored/coordinated  data  collection 
efforts.  TAEG  personnel  also  conducted  interviews  at  the  schools. 

Most  of  the  individuals  who  functioned  as  interviewers  participated 
in  a  2;?  nour  training  session  conducted  by  TAEG  at  each  school.  This  was 
given  to  promote  s tandardi za t. ion  (and,  thereby,  enhance  reliability)  of  the 
interview  procedures.  The  training  featured  a  videotape  of  a  "model"  inter¬ 
view  accompanied  by  a  verbal  explanation/discussion  of  the  desired  Interview 
Procedures  and  use  of  data  collection  instruments.  In  several  instances, 
designated  personnel  also  observed  TAFG  staff  conduct  interviews.  In  some 
rases,  owing  to  local  command  needs,  individuals  other  than  those  "trained" 
were  assigned  to  collect,  data  from  the  advanced  school  students.  All  indi¬ 
viduals,  however,  did  receive  an  interview  kit  which  contained  detailed 
instructions  for  conducting  interviews. 

Selecting  ana  scheduling  of  individuals  for  interviews  were  accomplished 
at  the  local  level.  Practices  emp'oyed  varied  at  different  locations 
because  of  different  administrative  procedures  affecting  availability  of 
the  advanced  students.  *t  some  NAVEDTRACOM  locations.  Background  Data 
Sheets  were  distributed  and  completed  durinq  class  time;  at  others,  they 
were  completed  by  students  as  part  of  a  school's  routine  check-in  procedure. 
Most  "qualified"  individuals  (i.e.,  had  observed  "A"  school  graduate  fleet 
performance  for  the  specified  period  and  length  of  time)  first  completed 
part  A  of  the  Feedback  Data  Form  and  were  then  scheduled  for  interviews  at 
a  later  date.  For  the  most  Dart,  interviews  were  conducted  on  a  time- 
available  basis  during  the  student's  stay  at  the  advanced  school.  The 
schools  were  encouraged,  however,  ~o  conduct  interviews  with  the  students 
as  soon  as  possible  after  their  arrival  to  avoid  any  possible  "biasing" 
effects  that  could  occur  from  continuing  exposure  to  Training  Command 
concerns,  attitudes,  or  "problems." 
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For  the  MS  portion  of  the  program,  feedback  data  were  collected  from 
school  attendees  at  the  SERVSCOl.COM,  San  Diego,  and  at  the  FLETRACEN, 

Norfolk.  Data  were  also  collected  from  fleet  MS  personnel  at  the  Recruit 
Training  Center,  Orlando,  and  within  East  and  West  Coast  fleet  units.  At 
the  FLETRACEN,  data  were  collected  from  "C"  school  students  on  a  time- 
available  basis.  All  interviews  at  the  FLETRACEN  were  conducted  by  the 
Curriculum  Instructional  Standards  Officer.  TAEG  personnel  interviewed 
MSs  at  Orlando.  These  MSs  were  scheduled  by  their  command.  Interviews  to 
collect  data  from  East  Coast  fleet  ships  and  shore  units  were  conducted  by 
the  Navy  Food  Management  Team  (NAVFOODMGTM)  at  Norfolk.  The  NAVFOODMGTM 
at  San  Diego  interviewed  MSs  at  West  Coast  activities.  All  interviews  by 
NAVFOODMGTM  personnel  were  conducted  during  routine  assistance  visits  to 
fleet  activities.  On  both  coasts,  interviews  were  conducted  on  a  time- 
available,  noninterfering  basis  by  the  NAVFOODMGTMs.  At  all  fleet  activities 
(including  Orlando),  parts  A  and  B  of  the  Feedback  Data  Form  were  completed 
simultaneously. 

DATA  PROCESSING  AND  ANALYSIS 

Data  collected  by  NAVEDTRACOM  personnel  within  the  schools  (including  ' 
Background  Data  Forms  completed  by  individuals  who  had  not  observed  recent 
graduate  performance  for  the  required  length  of  time)  were  mailed  to 
TAEG.  Data  collected  by  the  NAVFOODMGTMs  (for  MSs)  were  also  mailed. 

Within  TAEG,  all  data  were  recorded  on  worksheet  forms  devised  to  facili¬ 
tate  data  summarization  and  analysis.  (Separate  sets  of  forms  were,  of 
course,  used  for  each  rating/school.) 

Background  data  were  recorded  on  large  worksheet  forms  devised  for 
this  purpose.  Data  from  part  A  and  part  B  were  recorded  on  "Presentation 
of  Data"  worksheets.  One  worksneet  form  was  used  to  record  all  interview 
data  for  each  given  task.  Tabulations  were  made  from  the  worksheet  entries 
of  the  number  of  times  tasks  were  judged  by  interviewees  as  "Do  With 
Ease,"  "Do  With  Difficulty,"  or  "Don't  Do."  In  addition,  counts  were  made 
of  the  number  of  times  each  Reason  Code  was  chosen  for  tasks  categorized 
as  "Do  With  Difficulty"  or  "Don't  Do."  Comments  made  by  interviewees 
pertaining  to  each  task--either  to  clarify  why  particular  Reason  Codes 
were  selected  or  to  add  amplifying  information--were  also  recorded  on 
these  forms. 

Worksheet  data  were  used  to  determine  the  degree  to  which  recent 
graduates  were  utilized  to  perform  the  job  tasks  of  a  rating  at  activities 
represented  by  the  interviewees.  This  was  expressed  as  the  percentage  of 
interviewees  who  reported  that  the  graduates  were  used  at  fleet  units  to 
perform  each  of  the  various  job  tasks.  Utilization  percentages  were 
calculated  by  dividing  the  sum  of  all  "Do  With  Ease"  and  "Do  With  Diffi¬ 
culty"  responses  for  a  task  by  the  total  number  of  responses  for  that  task 
and  multiplying  the  quotient  by  100.  These  "utilization  percentages" 
reflect  relevancy  of  school  training  for  the  graduates'  fleet  jobs. 

The  worksheet  data  were  also  used  to  determine  the  degree  of  difficulty 
(conversely,  ease)  with  which  graduates  performed  each  of  the  various  job 
tasks  of  a  rating.  The  percent  of  respondents  (interviewees)  who  thought 
the  graduates  they  observed  did  a  task  with  difficulty  was  derived  by 
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dividing  the  number  of  "Do  With  Difficulty"  selections  for  each  task  by 
the  sum  of  all  "Do  With  Ease"  and  "Do  With  Difficulty"  responses  for  a 
task  and  multiplying  the  quotient  by  100.  (Computationally,  100  minus  the 
"Do  With  Difficulty"  percentage  gives  the  "Do  With  Ease"  percentage  for  a 
task.)  These  "percent  difficulty"  values  provide  indications  of  areas  where 
school  training  for  the  performance  of  given  tasks  may  be  deficient. 

SPECIAL  ANALYSES.  For  two  of  the  schools  (MS  and  AV),  opportunities  were 
made  to  evaluate/assess  the  equivalence  of  feedback  data  obtained  from 
individuals  in  different  assignment  categories. 

For  the  MS  rating,  feedback  data  collected  in  the  schools  were  statis¬ 
tically  compared  to  data  collected  within  the  fleet.  This  comparison  was 
important  to  the  overall  program  since  it  bears  directly  on  the  issue  of 
the  validity  of  feedback  from  the  school  source.  Findings  that  school 
data  are  equivalent  to  fleet  data  means  that  they  should  lead  to  essen¬ 
tially  the  same  conclusions  about  a  training  program.  Therefore,  data 
from  this  school  source  can  substitute  for  feedback  from  the  fleet. 

For  the  avionics  ratings  (AT,  AQ,  AX),  data  were  collected  from 
sufficient  numbers  of  IT  school  students  and  "C"  school  students  to  permit 
a  comparison  of  feedback  from  these  two  sources.  Petty  officers  attending 
IT  schools  are  typically  more  senior  than  those  attending  "C"  schools. 

Hence,  their  views  may  differ  from  the  "C"  school  students'  views.  Accord¬ 
ingly,  it  was  important  to  determine  if  there  were  any  substantial  differences 
in  the  feedback  data  from  these  two  groups  of  interviewees.  Findings  that 
the  data  were  equivalent  would  mean  that  data  could  be  gathered  indiscrim¬ 
inately  from  either  group  and  be  used  separately,  or  mixed,  with  confidence 
that  either  set/source  of  data  would  lead  to  essentially  the  same  conclusions 
about  the  job  performance  of  recent  graduates.  (Findings  that  the  data 
were  not  equivalent  in  this  case  would  provide  no  information  concerning 
which  source  was  the  most  desirable  for  obtaining  feedback.)  This  informa¬ 
tion  was  considered  important  to  the  overall  program  since,  with  few 
exceptions,  the  number  of  individuals  assigned  annually  to  IT  school  from 
a  particular  rating  is  relatively  small  while  a  fairly  substantial  number 
attend  the  "C"  schools. 

For  all  comparisons,  data  equivalence  was  assessed  through  the  use  of 
the  Pearson-Product  moment  correlation  technique  (see  ref.  5).  This 
statistic  yields  a  coefficient  of  correlation  (r)  which  indicates  numerically 
the  degree  of  relationship  between  two  sets  of  variables.  Correlation 
coefficients  may  take  on  values  ranging  from  0  to  plus  or  minus  1.  High 
correlation  coefficients  indicate  that  distributions  of  ratings/values  are 
similar.  Correlation  does  not  address  questions  of  similarity  in  magnitude 
(e.g.,  whether  these  are  significant  differences  in  the  average  values  of 
variables).  High  positive  correlations  between  appropriate  distributions 
of  ratings/values  obtained  from  different  sources  would  support  conclusions 
that  the  sources  provide  equivalent  data. 

METHOD  EVALUATION 

To  provide  a  partial  basis  for  evaluating  the  structured  interview 
method  used,  information  was  desired  from  school  staff  concerning  the 
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value  or  usefulness  for  curriculum  review  of  various  aspects  of  the  data. 

To  obtain  this  information,  all  data  for  each  rating  were  critically 
reviewed  by  subject  matter  experts  (SME)  at  the  appropriate  schools. 

At  the  completion  of  data  collection  concerning  each  school,  completed 
"Presentation  of  Data"  worksheets  for  the  job  tasks  evaluated  for  a  rating 
were  sent  to  that  school.  Copies  of  summary  data  reflecting  graduate 
utilization  to  perform  school-trained  tasks  and  difficulty  of  task  per¬ 
formance,  plus  all  general  comments  made  by  interviewees  concerning  changes/ 
improvements  to  the  "A"  school's  training,  were  also  transmitted. 

At  each  school,  five  staff  members  reviewed  the  data.  Each  SME 
completed,  independently  of  the  others,  one  "Usefulness  of  Data"  worksheet 
for  each  task.  These  worksheets  were  especially  devised  for  this  purpose. 
SMEs  recorded  their  opinions  on  the  worksheets  about  the  usefulness  of 
sorting  the  job  tasks  into  three  categories,  usefulness  of  the  reason 
codes  and  interviewee  comments,  and  about  the  overall  usefulness  of  the 
data.  The  rating  scale  allowed  three  choices  for  the  assessment:  "Not 
Helpful,"  "Of  Some  Help,"  "Very  Helpful."  The  SMEs  also  assessed  the  data 
for  their  contribution  to  training  problem  identification. 

The  completed  "Usefulness  of  Data"  worksheets  were  returned  by  mail 
from  the  MS  and  MR  schools  and  collected  during  visits  to  the  other  schools. 
At  these  other  schools,  working  meetings  were  held  between  TAEG  and  school 
staff  to  discuss  data  value  and  usefulness.  In  all  cases,  summaries  of 
the  SME  data  were  prepared  to  reflect  collective  opinions. 
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SECTION  III 

FINDINGS  AND  DISCUSSION 

Findings  of  the  program  which  bear  on  the  issues  of  feasibility  and 
desirability  of  obtaining  training  feedback  from  petty  officer  students  within 
the  NAVEDTRACOM  are  presented  below.  In  this  section,  summary  data  obtained 
over  all  six  schools  are  used  where  necessary  to  support  particular  findings  and 
to  facilitate  discussions  concerning  data  use  and  interpretation.  However, 
detailed  evaluation  data  concerning  specific  aspects  of  a  particular  school 
curriculum  are  not  reported.  This  information  can  be  obtained  from  the  individual 
school  evaluation  reports  (refs.  2,  6,  7,  10,  11,  and  12).  Information  gathered 
during  the  program  concerning  the  value  and  usefulness  for  curriculum  review  of 
the  method  used  is  also  presented. 

FEASIBILITY  ISSUE 

Findings  concerning  aspects  of  the  feasibility  of  collecting  training 
feedback  within  NAVEDTRACOM  schools  are  given  below. 

TIME  TO  COMPLETE  INTERVIEWS.  Time  to  collect  data  bears  directly  on  the  issue 
of  feasibility  of  collecting  data  within  the  school  environment.  This  is 
important  both  for  its  implications  for  "lost"  class  time  for  the  interviewees 
and  also  for  the  time  which  school  staff  lose  from  other  duties  while  conducting 
interviews. 

The  average  times  required  to  complete  part  A  and  part  B  of  the  "interview" 
procedure  are  shown  in  table  1.  No  appreciable  staff  time  was  involved  in  part 
A  since  the  interviewee  completed  this  independently.  Part  B  did  involve  both 
interviewer  and  interviewee  time.  The  total  time  shown  in  the  table  is  simply 
the  sum  of  the  two  means  which  reflects  an  average  time  for  a  student  to  complete 
both  parts  of  the  interview  procedure. 

TABLE  1.  AVERAGE  TIMES  (MINUTES)  TO  COMPLETE  INTERVIEWS 


Rating 

Average  Completion  Times 

Part  A  Part  B 

No.  of  Tasks 

Total  Average 

AD 

10 

25 

50 

35 

MR 

14 

42 

63 

56 

EN 

16 

21 

45 

37 

MS 

23 

38 

83 

61 

AV1 

17 

26 

51 

43 

FT 

14 

36 

31 

50 

^Includes  AT,  AQ,  AX  ratings 
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In  the  worst  cases  (MR,  MS),  the  average  time  required  for  a  respondent 
(interviewee)  to  complete  the  procedure  was  approximately  1  hour.  Staff  time  to 
conduct  the  interviews  (part  B)  did  not  exceed  45  minutes.  This  met  the  expec¬ 
tations/desires  of  school  commands  to  limit  interviews  to  approximately  1  hour. 
Accordingly,  there  were  no  suggestions  of  undue  burdens  being  placed  on  the 
schools  by  virtue  of  regularly  assigned  school  staff  taking  time  off  from  other 
duties  to  conduct  the  interviews.  Over  all  six  schools,  the  average  student 
time  to  address/discuss  any  one  job  task  was  64  seconds.  Thus,  for  future  data 
collection  with  both  parts  of  the  procedure  being  completed  simultaneously, 
attention  could  be  focused  on  at  least  50  job  tasks  within  a  1  hour  period. 

AVAILABILITY  OF  QUALIFIED  JUDGES.  A  second  aspect  of  the  feasibility  issue 
concerned  the  availability  within  the  schools  of  qualified  judges  of  "A" 
school  graduate  performance.  For  the  program  conducted,  a  qualified  respondent 
was  arbitrarily  defined  as  an  individual  ("C"  school  or  IT  school  student)  who 
had  within  the  immediately  preceding  year  observed  the  fleet  job  performance 
of  at  least  one  recent  (i.e.,  3-6  months  after  graduation)  "A"  school  graduate 
for  a  minimum  of  3  months. 

Table  2  shows  the  number  of  individuals  reported  attending  IT  and  "C" 
schools  during  periods  of  the  study.  It  also  shows  the  number  (and  percentage) 
in  each  rating  who  met  the  eligibility  criterion.  According  to  the  data 
in  the  table  (which  was  reported  by  the  participating  schools),  it  can  be 
expected  that  approximately  45  to  80  percent  of  IT  students  in  a  rating  and  50 
to  75  percent  of  those  selected  for  "C"  school  can  provide  feedback  on  "A" 
school  training.  The  proportion  of  MS  "C"  school  students  at  the  SERVSC0LC0M, 
San  Diego,  who  reported  that  they  had  not  observed  recent  "A"  school  graduate 
performance  is  unusually  high.  The  exact  reasons  for  this  are  unknown.  In 
view  of  the  Norfolk  "C"  school  data,  however,  and  data  from  the  other  ratings, 
it  is  anticipated  that  future  samples  of  MS  "C"  school  students  at  San  Diego 
would  show  approximately  the  same  values  as  the  other  schools. 

The  final  column  of  table  2  shows  the  percentage  (in  parentheses)  of 
advanced  students  (i.e.,  IT  and  "C"  school  students  combined)  who  were  con¬ 
sidered  to  be  qualified  judges  of  "A"  school  graduate  performance.  These 
data  suggest  that  approximately  6  of  every  10  (median  percentage  equals  64) 
petty  officers  ordered  to  the  NAVEDTRACOM  for  advanced  training  can  provide 
feedback  information  concerning  an  "A"  school  graduate's  performance.  Thus, 
for  future  data  collection  efforts,  the  number  of  potential  interviewees  can 
be  estimated  as  60  percent  of  the  A0B  or  projected  student  input. 

One  additional  point  is  relevant  to  the  question  of  future  data  collection 
from  IT  and/or  "C"  school  students.  This  concerns  the  number  of  individuals 
who  should  be  interviewed  to  obtain  reliable  feedback  data  concerning  "A"  school 
training.  The  TAEG  study  program  was  more  directly  aimed  at  the  evaluation 
of  the  methodology  developed  than  at  evaluation  of  an  "A"  school's  training. 

A  greater  number  of  interviews  were  needed  for  reliable  assessment  of  the  method 
than  would  normally  be  needed  for  curriculum  evaluation  purposes.  While  no 
universal  rule  can  be  given  regarding  the  number  of  individuals  to  interview 
to  obtain  reliable  feedback  data,  a  convenient  rule  of  thumb  is  to  continue 
interviewing  until  no  new  information  is  being  obtained.  That  is,  when  inter¬ 
viewee  comments,  for  example,  become  highly  redundant  to  information  already 
obtained,  it  can  be  considered  that  further  data  collection  is  unnecessary. 
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TABLE  2.  DISTRIBUTION  OF  PETTY  OFFICERS  INTERVIEWED 
WITHIN  THE  NAVEDTRACOM 


Data  Collection 

Period 

Number 

Attending 

Number 
(and  %) 
Interviewed 

Total  No. 

(and  %) 
Interviewed 

Rating 

Start 

End  IT 

School 

C 

School 

IT 

School 

C  School 

AD 

5/22/78 

11/28/78 

14 

56 

9 

(64) 

35  (63) 

44  (63) 

MR 

6/21/78 

12/18/78 

8 

59 

6 

(75) 

32  (54) 

38  (57) 

EN 

7/26/78 

5/14/79 

11 

84 

5 

(45) 

46  (55) 

51  (54) 

MS1 

6/21/783 

2/27/79 

12 

128 

10 

(83) 

38  (30) 

48  (34) 

6/21 /78b 

11/17/78 

- 

26 

- 

17  (65) 

17  (65) 

AV2 

5/1 1/78C 

1/26/79 

38 

- 

30 

(80) 

- 

30  (80) 

1 l/08/78d 

1/26/79 

- 

38 

- 

28  (74) 

28  (74) 

FT 

7/25/78 

8/31/79 

18 

14 

14 

(78) 

11  (79) 

25  (78) 

^Does 

not  include 

82  MSs  interviewed 

in 

fleet 

billets 

aSERVSC0LC0M,  San  Diego 
bFLETRACEN,  Norfolk 


2 

Includes  AT,  AQ,  AX  ratings 

CIT  students 

d"C"  school  students. 

Equivalence  of  IT  and  School  Student  Data.  During  the  program,  a  logical 
question  arose  concerning  the  equivalence  of  IT  student  and  "C"  school  student 
feedback  data.  This  issue  has  direct  implications  for  the  number  of  "qualified" 
judges  available  within  the  Command.  The  correlation  obtained  between  AV  IT 
and  "C"  school  students'  data  for  percent  utilization  of  recent  AV  A1  graduates 
to  perform  surveyed  job  tasks  was  .86.  The  correlation  between  the  data  for  the 
percent  reporting  graduate  ease/difficulty  of  task  accomplishment  was  .71.  Both 
correlations  are  statistically  significant.  Both  reflect  a  high  degree  of 
relationship  between  the  data  from  the  two  groups  and  support  a  conclusion  that 
feedback  data  from  either  group  are  interchangeable.  Thus,  feedback  data  of  the 
form  collected  during  this  program  whether  obtained  from  IT  or  "C"  school  students 
should  lead  to  essentially  the  same  conclusions  about  school  training  even  though 
the  IT  school  students  were  considerably  more  senior  (i.e.,  higher  rated,  greater 
time  in  service)  than  the  "C"  school  students.  This  finding  is  important  since 
the  majority  of  petty  officer  students  within  the  NAVEDTRACOM  are  assigned  to 
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"C"-level  courses.  Although  this  assessment  could  only  be  made  for  the  Avionics  7 

Technician,  it  is  believed  that  similar  results  would  be  obtained  for  other 

ratings.  I 

DESIRABILITY  ISSUE 

The  most  important  consideration  underlying  the  desirability  of  obtaining 
training  feedback  within  the  NAVEDTRACOM  is  the  validity  of  the  data  obtained 
from  the  advanced  school  students.  In  this  context,  validity  refers  to  the 
equivalence  of  data  from  school  groups  and  current  fleet  users  of  the  "A"  school 
graduates.  During  the  program,  data  validity  was  (1)  inferred  based  on  examina¬ 
tion  of  the  interviewees'  background  characteristics  and  (2)  assessed  statisti¬ 
cally  for  the  MS  rating. 

VALIDITY  OF  SCHOOL  FEEDBACK  DATA.  Feedback  obtained  within  the  NAVEDTRACOM  was 
assumed  to  be  valid  for  the  reasons  presented  below.  At  all  six  schools,  the 
petty  officers  int  ."viewed  had  only  recently  returned  from  fleet  billets  (i.e., 
within  the  previous  6  to  8  weeks).  They  reported  to  school  billets  from  broad, 
diverse  groups  of  Navy  units  typical  of  those  to  which  the  respective  "A"  school 
graduates  are  assigned.  As  a  group,  the  interviewees  had  a  wide  range  and 
breadth  of  experience  in  their  respective  ratings.  All  had  recent  opportunity 
to  observe  the  fleet  job  performance  of  "A"  school  graduates.  Because  of  these 
factors,  it  is  believed  that  they  adequately  represented  fleet  users  of  the  "A" 
school  graduates  (i.e.,  they  constituted  representative  samples  of  the  larger  fleet 
user  populations).  Accordingly,  it  was  assumed  that  the  data  obtained  from  the 
school  groups  would  be  equivalent  to  data  from  individuals  still  serving  in  fleet 
billets. 

For  the  MS  rating,  opportunity  was  created  to  test  this  conclusion  statisti¬ 
cally.  As  mentioned  previously,  feedback  data  were  collected  simultaneously 
from  MSs  attending  NAVEDTRACOM  schools  and  MSs  currently  in  fleet  billets. 

Eighty-three  job  tasks  were  evaluated  by  each  group.  Correlational  analyses 
were  performed  on  the  data  from  the  two  groups  of  MSs.  The  correlation  between 
graduate  utilization  proportions  reported  by  82  fleet  MS  petty  officers  and 
those  reported  by  65  petty  officers  attending  IT  and  "C"  schools  was  .91.  The 
correlation  between  fleet  reported  performance  difficulty  proportions  and  school 
source  reports  of  performance  difficulty  was  .87.  These  high  correlations 
indicate  that  the  two  sources  do  provide  equivalent  information. 

While  the  above  finding  is  specific  to  the  MS  rating  (and  the  type  of  data 
collected),  there  are  no  reasons  apparent  to  suggest  that  the  school  source 
would  provide  training  feedback  information  different  from  the  fleet  source  for 
most,  if  not  all,  other  ratings  as  well.  Exclusive  future  reliance  on  the 
school  source  for  feedback  information  does  not  seem  warranted,  however,  since 
there  may  be  only  relatively  small  numbers  of  advanced  students  (who  have  observed 
"A"  school  graduate  performance)  available  within  the  NAVEDTRACOM  at  any  given 
point  in  time.  Depending  upon  the  urgency  of  feedback  needs,  data  from  fleet 
personnel  will  probably  still  be  desirable. 

USE  AND  INTERPRETATION  OF  FEEDBACK  DATA 

Information  concerning  the  use  and  interpretation  of  graduate  task  utilization 
and  task  performance  difficulty/ease  data  is  given  below.  Summaries  of  data 
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obtained  during  the  program  are  used  to  facilitate  the  discussion.  These  surnnaries 
should  not  be  used  to  formulate  conclusions  about  a  particular  school's  training. 
The  detailed  information  presented  in  the  individual  reports  (refs.  2,  6,  7,  10, 

11,  and  12)  is  more  suitable  for  this  purpose. 

UTILIZATION  DATA.  Table  3  shows  the  number  of  tasks  for  each  rating  falling 
into  various  percent  utilization  categories  and  the  number  of  tasks  evaluated 
for  each  rating.  The  table  shows,  for  the  units  represented  by  the  individuals 
interviewed,  the  percentage(s )  of  typical,  recent  "A"  school  graduates  who 
were  used  to  perform  job  tasks  for  which  they  received  training  at  the  school. 

Two  examples  of  how  to  read  table  3  are: 

•  91  to  100  percent  of  typical,  recent  AD  "A"  school  graduates  performed 
8  of  50  school-trained  tasks  at  their  assigned  units 

•  81  to  90  percent  of  typical,  recent  MS  "A"  school  graduates  were  used 
to  perform  25  of  the  school-trained  tasks  evaluated  by  the  MS  inter¬ 
viewees. 

TABLE  3.  SUMMARY  OF  "A"  SCHOOL  GRADUATES'  FLEET 

UTILIZATION  TO  PERFORM  SCHOOL-TRAINED  TASKS 
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The  cumulative  percent  utilization  values,  shown  in  parentheses  in  the 
table,  can  be  used  in  various  ways  to  assess  the  relevancy  of  school  training 
for  operational  job  requirements.  For  example,  51-60  percent  of  the  typical  "A" 
School  graduates  were  used  at  the  units  represented  to  perform: 

•  90  percent  (i.e,,  45  of  50)  of  the  job  tasks  considered  by 
the  AD  rating 

•  100  percent  of  the  63  job  tasks  evaluated  by  MRs 

•  78  percent  of  the  EN  job  tasks 

•  80  percent  of  the  MS  job  tasks 

•  76  percent  of  the  AV  job  tasks 

•  52  percent  of  the  FT  job  tasks. 

Overall,  the  cumulative  percentages  in  the  table  indicate  that  utilization 
of  graduates  to  perform  school -trained  job  tasks  is  relatively  high  for  all 
the  ratings  surveyed.  The  data  suggest  that  the  most  job  relevant  curriculum 
(as  reflected  by  the  tasks  evaluated  for  the  schools  surveyed)  is  the  MR 
curriculum.  At  least  80  percent  of  typical  recent  graduates  are  used  at  the 
units  represented  by  the  interviewees  to  perform  more  than  half  (56  percent) 
of  the  tasks  trained  at  the  "A"  school.  Fifty  percent  or  more  of  the  MR 
graduates  perform  all  of  the  school-trained  tasks  at  the  units. 

Aggregated  as  they  are  for  this  report,  percent  utilization  data  could 
be  useful  for  management  information  purposes.  For  example,  utilization 
data  could  be  used  to  compare  different  schools  in  terms  of  the  responsiveness 
or  relevance  of  their  training  to  operational  fleet  job  requirements/expecta¬ 
tions  concerning  graduates.  At  the  individual  school  level,  the  utilization 
value  data  are  useful  for  suggesting  specific  tasks  for  which  school  training 
may  be  either  totally  eliminated  or  training  emphases  changed.  Decisions  of 
this  type,  however,  also  require  additional  knowledge  concerning  reasons  for 
nonutilization  of  the  graduates. 

The  interview  procedure  was  designed  to  gather  some  of  the  necessary 
additional  information.  The  reason  codes  used  with  "Don't  Do"  choices  provided 
one  source  of  this  information.  Interviewee  comments  provided  another.  Table 
4  lists  the  predominant  reasons  given  (through  the  mechanism  of  reason  code 
selection)  for  nonutilization  of  graduates.  The  primary  reason  given  across  all 
ratings  was  that  the  task  was  not  required  at  that  particular  unit.  Still,  more 
information  is  needed,  however,  concerning  why  such  tasks  are  not  performed.  In 
this  case,  interviewee  comments  should  be  examined  to  determine  if  training 
emphases  can  be  changed.  It  may  be  found,  for  example,  that  certain  job  tasks 
may  no  longer  be  requirements  of  a  rating. 

PERFORMANCE  DIFFICULTY/EASE  DATA.  While  the  percent  utilization  values  have 
implications  for  relevance  of  school  training  (i.e.,  "are  the  right  things  being 
taught?"),  the  percent  difficulty  values  reflect  on  the  quality  of  school  training 
for  the  fleet  job.  These  values  relate  to  the  question  of  "How  well  the  school 
prepares  individuals  for  the  operational  job." 
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TABLE  4.  SUMMARY  OF  REASONS  FOR  NONUTILIZATION  OF  GRADUATES 
TO  PERFORM  SCHOOL  TRAINED  TASKS 


Rating 

Percent 

Code 

1* 

Time  Reason 

Selected 

2** 

AD 

54 

39 

MR 

41 

38 

EN 

79 

12 

MS 

38 

24 

AV1 

69 

27 

FT 

37 

47 

*  Reason  Code  1--Task  not  performed  on  my  ship/aircraft/station/system 
**  Reason  Code  2--Task  not  expected  of  someone  in  recent  graduates's  rate 
or  level  of  experience 


includes  AT,  AQ,  AX  ratings 

Table  5  presents  data  showing  the  ease/difficulty  with  which  typical, 
recent  "A"  school  graduates  were  reported  to  perform  job  tasks  at  the  units 
represented  by  the  interviewees.  The  table  shows  the  percent  of  recent  graduates 
(first  column)  who  perform  a  given  number  of  tasks  with  ease  (cell  entries),  and 
the  percent  of  recent  graduates  who  perform  the  same  tasks  with  difficulty  (last 
column).  The  table  shows,  for  example,  that  71-80  percent  of  typical,  recent  MR 
"A"  school  graduates  perform  20  job  tasks  of  the  rating  without  difficulty  ( i . e . , 
with  ease)  and  that  20-29  percent  of  the  graduates  have  difficulty  performing 
these  same  tasks.  Examination  of  the  data  in  table  5  shows  that,  overall, 
typical  "A"  school  graduates  are  reported  to  have  difficulty  performing  many  of 
the  tasks  for  which  they  receivec  training  at  the  schools. 

The  percent  difficulty  values  are  useful  for  making  preliminary  decisions 
concerning  the  possible  existence  of  training  "problems."  For  example,  if  1  of 
every  5  graduates  (20  percent  difficulty  value)  is  reported  to  have  difficulty 
performing  a  particular  task,  additional  information  should  be  obtained  to 
determine  if  training  requires  attention.  Two  sources  of  such  additional 
information  are  available  from  the  interview  procedure:  reason  codes  and  inter¬ 
viewee  comments. 

For  the  program  conducted,  "reasons"  for  performance  difficulties  were 
represented  by  letter  codes  ("A,"  "B,"  "C,"  etc.).  Some  of  these  were  aimed  at 
discovering  which  performance  difficulties  may  not  be  attributable  to  and, 
hence,  not  easily  correctable  by  school  training  (e.g.,  equipment  operating 
peculiarities,  difficult  access  to  equipment  for  task  performance  and  other 
peculiarities  of  a  specific  work  environment).  A  second  source  of  information 
concerning  reasons  for  performance  difficulties  was  the  comments  made  about  the 
specific  nature  of  the  performance  difficulties. 
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TABLE  5.  SUMMARY  OF  GRADUATE  EASE  OF  TASK  PERFORMANCE  DATA 


Percent 

Perform  With 
Ease  Values 

AD 

Number  of  Tasks  in  Each 
Category  for  Rating 

MR  EN  MS 

AV1 

FT 

Percent  Perform 

With  Difficulty 
Values 

91-100 

4 

3 

8 

7 

4 

0 

0-9 

81-90 

5 

14 

12 

17 

5 

2 

10-19 

71-80 

4 

20 

17 

24 

8 

4 

20-29 

61-70 

6 

8 

3 

18 

9 

3 

30-39 

51-60 

11 

5 

3 

9 

5 

1 

40-49 

41-50 

7 

8 

2 

4 

12 

2 

50-59 

31-40 

7 

2 

4 

5 

10 

60-69 

21-30 

2 

3 

3 

H 

70-79 

11-20 

1 

4 

80-89 

3 

1 

90-100 

Number  of 

Tasks  Evaluated 

50 

63 

45 

83 

51 

31 

includes  AT,  AQ,  AX  ratings 


Taken  together,  these  two  sources  provided  information  to  assess  which  tasks 
should/could  be  addressed  in  terms  of  remedial  training  and  also  the  "type"  of 
remedial  action  indicated  to  correct  subsequent  on  the  job  performance  diffi¬ 
culties.  Reasons  given  for  performance  difficulties  were  highly  variable 
across  the  ratings  surveyed.  Consequently,  they  cannot  be  succinctly  summarized 
here.  This  information  is  contained,  however,  in  the  individual  school  reports 
(refs.  2,  6,  7,  10,  11,  and  12). 

METHOD  EVALUATION 

As  mentioned  previously  all  data  obtained  from  the  interviews  were  reviewed 
by  five  staff  SMEs  at  each  of  the  six  schools.  This  review  was  requested  by  the 
TAEG  to  provide  an  evaluation  of  the  structured  interview  method  in  terms  of 
the  usefulness  of  the  data  for  curriculum  review  and  its  relevance  for 
identifying  training  problems. 


JL 
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DATA  USEFULNESS.  Table  6  summarizes  SMEs  opinions  across  the  six  "A"  schools 
concerning  the  usefulness  of  the  interview  data  for  curriculum  review. 

The  entries  in  the  cells  are  average  (mean)  "usefulness"  values.  These  were 
computed  by  assigning: 

•  "0"  to  "not  helpful"  choices 

•  "1"  to  "of  some  help"  choices 

•  "2"  to  "very  helpful"  choices. 

The  means  were  derived  by  dividing  the  sum  of  the  numerical  values  by  the  number 
of  tasks  evaluated  for  a  rating. 


TABLE  6.  SUMMARY  OF  SME  OPINION  OF  USEFULNESS  OF  THE  INTERVIEW  DATA 


Mean 

Usefulness  Ratings 

AD 

MR 

EN 

MS 

FT 

Usefulness  of  "Do 
with  Ease,"  "Don't 

Do"  and  "Do  with 

Difficulty"  Figures 

1.94 

1.11 

1.98 

1.0 

1.14 

1.19 

Usefulness  of  Reason 

Codes  and  Figures 

1.18 

1.03 

1.56 

.99 

.88 

1.10 

Usefulness  of  Comments 
(Clear,  precise,  etc.) 

1.16 

1.06 

1.16 

VO 

PO 

1.18 

1.10 

Overall  Usefulness  of 

Data 

1.42 

1.03 

1.84 

1.0 

1.06 

1.13 

includes  AT,  AQ,  AX  ratings 


As  is  evident  from  the  table,  school  SMEs  generally  considered  the  data 
to  be  helpful  for  curriculum  review.  Reason  codes  which  provided  reasons  for 
graduate  difficulties  or  for  nonperformance  of  tasks  were  also  generally 
considered  helpful.  They  were  considered  to  be  of  most  use,  however,  when 
they  were  coupled  with  amplifying  comments  which  explained  the  reasons  for  their 
selection. 

Interviewee  comments  which  provided  detailed  information  concerning  the 
specific  nature  of  graduate  performance  difficulties  were  generally  considered 
by  all  SMEs  to  be  the  most  useful  feature  of  the  data.  However,  the  technical 
content  and  clarity  of  the  corments  varied  considerably.  This  should  probably 
be  attributed  to  the  relative  inexperience  of  the  interviewers  rather  than  to 
any  inherent  defects  of  the  method.  As  experience  is  gained  with  interviewing 
procedures,  interviewers  usually  become  more  skillful  in  extracting  directly 
relevant,  detailed  information  and  in  stating  it  in  more  clear  and  concise  terms. 
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TRAINING  PROBLEM  IDENTIFICATION.  A  final  consideration  concerning  the  value 
of  any  feedback  data  collection  method  pertains  to  the  extent  to  which  obtained 
data  permit  identification  of  training  deficiencies.  Figure  1  summarizes  SMEs' 
opinions  across  the  six  schools  concerning  the  "relevance"  of  the  interview 
data  for  permitting  identification  of  training  "problems."  The  cells  show  the 
numbers  of  tasks  falling  into  each  category.  Placement  of  a  task  within  a  given 
category  was  either  by  SME  consensus  (AD,  EN,  AV,  FT)  or  agreement  of  at  least 
3  of  the  5  reviewers  at  a  school  (MS,  MR).  As  the  table  shows,  SMEs  identified 
a  substantial  number  of  tasks  as  representing  areas  of  training  deficiency. 

POSTNOTE 

For  future  data  collection  it  is  desirable  that  personnel  assigned 
interviewing  duties  have  better  (i.e.,  more)  training  and/or  greater 
procedural  experience  than  those  employed  in  this  program.  This  would 
undoubtedly  enhance  the  reliability  of  the  procedure  and  result  in  more 
direct,  concise  statements  concerning  the  nature  of  graduate  job  performance 
difficulties.  However,  the  overall  results  of  the  study  did  demonstrate  that 
interviewing  duties  could  be  shared  among  a  variety  of  school  personnel  who  are 
relatively  inexperienced  in  interviewing  techniques.  As  long  as  the  procedures 
are  followed  in  a  reasonable  way,  useful  data  can  be  obtained. 

SUMMARY  OF  FINDINGS 

During  this  study  program,  training  feedback  data  concerning  the  curricula 
of  six  "A"  schools/courses  were  collected  within  the  NAVEDTRACOM  from  281  petty 
officers  representing  eight  different  ratings.  Data  were  also  collected  from 
82  MS  petty  officers  still  serving  in  fleet  billets.  A  structured  interview 
method  was  used  for  data  collection.  Data  and  experience  gained  during  the 
program  were  used  to  assess  the  feasibility  and  desirability  of  obtaining 
feedback  data  within  the  school  environment  and  to  evaluate  the  utility  of 
the  structured  interview  method  for  this  purpose.  The  principal  findings  of 
the  program  which  bear  on  its  overall  objectives  are  summarized  below. 

1.  A  review  of  the  background  characteristics  of  petty  officers  attend¬ 
ing  advanced  schools  within  the  NAVEDTRACOM  and  the  recency  of  their  fleet 
assignments  led  to  the  conclusion  that  feedback  provided  by  them  would  not 
differ  substantially  from  feedback  that  could  be  gathered  from  their  counter¬ 
parts  still  serving  in  operational  fleet  billets.  Thus,  training  feedback 
from  the  school  petty  officer  groups  who  were  interviewed  during  the  study 
program  was  considered  to  be  valid. 

2.  The  structured  interview  method  yields  valuable  data  for  curriculum 
review.  The  data  are  useful  for  identifying  training  deficiencies  and  the 
nature  of  those  deficiencies. 

3.  Approximately  60  percent  of  the  petty  officers  assigned  to  advanced 
courses  within  the  NAVEDTRACOM  were  qualified  to  evaluate  the  fleet  job  per¬ 
formance  of  recent  "A"  school  graduates. 

4.  The  AV  portion  of  the  study  program  demonstrated  that  essentially 
the  same  information  about  "A"  school  graduate  fleet  job  performance  can  be 
obtained  from  either  IT  school  students  or  more  junior  "C"  school  students. 
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Rating  Six  Course 

Category  NAVEDTRACOM 

AD  MS  EN  AV  MR  FT  Average  (Median) 


Figure  1.  Summary  of  SMEs  Use  of  Data  to  Identify  Training  Problems 
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This  is  particularly  important  since  "C"  school  students  comprise  the  majority 
of  advanced  school  attendees. 

5.  The  MS  portion  of  the  study  demonstrated  statistically  that  feedback 
obtained  from  personnel  attending  advanced  schools  within  the  NAVEDTRACOM  was 
equivalent  to  feedback  obtained  directly  from  fleet  sources.  This  finding 
reinforces  the  assumption  made  under  paragraph  1  above.  The  MS  effort  also 
demonstrated  that  the  structured  interview  method  can  be  used  successfully  to 
collect  training  feedback  within  the  fleet. 

6.  No  "complaints"  were  voiced  concerning  the  use  of  school  facilities, 
staff  for  technical  assistance  and  interviewing,  or  student  time.  No  reasons 
became  evident  to  assume  that  use  of  the  method  within  the  schools  was 
undesirable  from  an  "Inconvenience/disruption"  of  routine  standpoint. 

7.  The  average  time  required  to  complete  Individual  Interviews  at  all 
six  schools  was  approximately  1  hour.  This  met  the  expressed  desires  of 
school  commands. 

8.  Interviewing  duties  can  be  shared  among  a  variety  of  school  personnel 
inexperienced  in  interviewing  techniques.  As  long  as  the  procedures  are 
followed  in  a  reasonable  way,  useful  data  can  be  obtained.  However,  better 
training  for  interviewers  would  undoubtedly  have  improved  the  quality  of 
comments  describing  the  specific  nature  of  training  problems. 
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SECTION  IV 

CONCLUSIONS  AND  RECOMMENDATIONS 


CONCLUSIONS 

Experience  gained  and  feedback  data  collected  during  this  work  program  lead 
to  the  conclusions  that: 

•  It  is  both  feasible  and  desirable  to  collect  training  feedback 
information  within  the  NAVEDTRACOM  advanced  school  context  from 
petty  officers  who  rotate  from  fleet  billets  to  attend  IT  or 
"C"-level  courses  In  their  ratings. 

•  The  structured  interview  procedure  yields  valuable  and  useful 
information  for  curriculum  evaluation. 

RECOMMENDATIONS 

It  is  recommended  that  the  structured  interview  method  assessed  during 
the  study  program  continue  to  be  used  to  collect  training  feedback.  The  method 
can  be  used  to  collect  information  concerning  the  specific  nature  of  any 
training  deficiencies  that  may  be  revealed  by  NAVEDTRACOM  Level  II  training 
appraisal  surveys,  or  to  conduct  Level  III  course  appraisals  (see  OPNAV  Instruc¬ 
tion  1540.50,  15  May  1979).  It  can  also  be  used  to  obtain  feedback  information 
within  the  schools  on  a  continuing,  routine  basis  when  a  course  is  not  scheduled 
for  formal  evaluation  by  the  command. 


31/32 


TAEG  Report  No.  92 


REFERENCES 

1 .  B11 inski ,  C.  R.  and  Saylor,  J.  C.  Training  Feedback  to  the  Navy  Storekeeper 

Class  "A"  School.  Research  Report  No.  SRR  72-1 4 .  January  1972.  Naval 
Personnel  and  Training  Research  Laboratory,  San  Diego,  CA. 

2.  Denton,  C.  F.  and  Hall ,  E.  R.  Evaluation  of  Machinery  Repairman  (MR)  A  School . 

TAEG  Technical  Memorandum  79-4.  August  1979”.  Tra  T  n 1  ng  An  a  1  y  sTs  and 
Evaluation  Group,  Orlando,  FL. 

3.  Dyer,  F.  N.,  Mew,  D.  V.,  and  Ryan,  L.  E.  Procedures  for  Questionnaire  Develop¬ 

ment  and  Use  in^  Navy  Training  Feedback.  TAEG  Report  No!  20.  October  1975. 
Training  Analysis  and  Evaluation  Group,  Orlando,  FL.  (AD  A018069) 

4.  Dyer,  F.  N. ,  Ryan,  L.  E. ,  and  Mew,  D.  V.  A_  Method  for  Obtaining  Post  Formal 

Training  Feedback;  Development  and  Validation.  TAEG  Report  No.  T9. 

May  1975.  Training  Analysis  and  Evaluation  Group,  Orlando,  FL  (AD  A011118) 

5.  Guilford,  J.  P.  Psychometric  Methods.  New  York:  McGraw-Hill,  Inc.  1954. 

6.  Hall,  E.  R.  and  Denton,  C.  F.  Evaluation  of  Mess  Management  Specialist  (MS) 

"A"  School  Training  by  Advanced  MS  NAVFDTRAfcOM~Students  and  by  Fleet  MS 
Personnel .  TAEG  Report  No.  76.  October  1979.  Training  Analysis  and 
Evaluation  Group,  Orlando,  FL.  (AD  A079558) 

7.  Hall,  E.  R.  and  Hughes,  H.  Jr.  Evaluation  of  Avionics  Technician  Class  A1 

Course.  TAEG  Technical  Memorandum  80-4.  April  19357  Training  Analysis 
and  Evaluation  Group,  Orlando,  FL. 

8.  Hall,  E.  R.,  Lam,  K.,  and  Bellomy,  S.  G.  Training  Effectiveness  Assessment 

Volume  I_:  Current  Military  Training  Evaluation  Programs .  TAEG  Report 
No.  39.  December  1976.  Training  Analysis  and  Evaluation  Group,  Orlando, 

FL.  (AD  A036517) 

9.  Hall,  E.  R.,  Rankin,  W.  C.,  and  Aagard,  J.  A.  Training  Effectiveness  Assessment 

Volume  II:  Problems ,  Concepts ,  and  Evaluation  Alternatives.  TAEG  Report 
No.  39.  December  1976.  Training  Analysis  and  Evaluation  Group,  Orlando,  FL. 
(AD  A036518) 

10.  Hughes,  H.  Jr.  and  Hall ,  E.  R.  Evaluation  of  Fire  Control  Technician  Class  "A" 

Phase  I  Course.  TAEG  Technical  Memorandum  80-5,  May  I960.  Training 
AnalysTs  and  Evaluation  Group,  Orlando,  FL. 

11.  Hughes,  H.  Jr.  and  Hall,  E.  R.  Evaluation  of  Enqineman  (EN)  Class  "A"  Course. 

TAEG  Technical  Memorandum  79-S.  October  19/9.  Training  AnalysTs  and 
Evaluation  Group,  Orlando,  FL. 

12.  Hughes,  H.  Jr.  and  Hall,  E.  R.  Evaluation  of  Aviation  Machinist's  Mate  (AD)  Al_ 

Course.  TAEG  Technical  Memorandum  79-17  June  1979"!  Training  AnalysTs 
and  Evaluation  Group,  Orlando,  FL. 

13.  Naval  Amphibious  School.  Fleet  Feedback  Information.  3  September  1975.  Naval 

Amphibious  Base,  Coronado,  CA. 


33 


TAEG  Report  No.  92 


REFERENCES  (continued) 

14.  Siegel,  A.  I.,  Schultz,  D.  G.,  and  Federman,  P.  Post-Training  Performance 

Criterion  Development  and  Application:  A  Matrix  Method  tor  the  Evaluation 
of  Training.  January  1961.  Contract  Nonr-2279(00) .  Personnel  and  Training 
Branch,  Office  of  Naval  Research,  Arlington,  VA. 


15.  Standlee,  L.  S.,  Bilinski,  C.  R.,  and  Saylor,  J.  C.  Procedures  for  Obtaining 

Training  Feedback  Relative  to  Electronics  Maintenance.  Report  No.  SRR  72-13. 
January  1972.  Raval  Personnel  and  Training  Research  Laboratory,  San  Diego, 
CA. 


34 


TAEG  Report  No.  92 


APPENDIX 

GUIDELINES  FOR  CONDUCTING  A  STRUCTURED  INTERVIEW 
FEEDBACK  DATA  COLLECTION  PROGRAM 


This  appendix  presents  guidelines  for  implementing  feedback  data  collection 
programs  using  the  structured  interview  method  assessed  during  the  TAEG  study. 
Recommendations  for  data  use  in  curriculum  evaluation/revision  are  also 
provided. 
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INTRODUCTION 

The  results  of  the  study  program  demonstrated  that  the  structured 
interview  procedure  can  be  used  to  obtain  training  feedback  information  which 
would  be  valuable  for  school  use  In  Identifying  training  deficiencies  and 
the  nature  of  those  deficiencies.  The  method  can  be  used  on  a  routine  basis 
to  collect  feedback  from  newly  reporting  petty  officers.  This  feedback  can 
be  used  whenever  needed  (1)  for  curriculum  review,  (2)  as  input  to  the  Annual 
Course  Review,  or  (3)  to  obtain  additional  detailed  information  on  training 
inadequacies  revealed  by  fleet  complaints  or  by  data  collected  via  the 
NAVEOTRACOM  Training  Appraisal  System.  Since  the  collection  of  feedback  data 
within  the  NAVEOTRACOM  does  not  Impact  or  impinge  on  fleet  resources  or 
otherwise  require  access  to  fleet  personnel,  data  can  be  collected  at  any 
time.  These  data  should  be  particularly  desirable/useful  during  those  years 
in  which  a  particular  course  is  not  scheduled  for  a  formal  evaluation  by 
CNET. 

The  remainder  of  this  appendix  presents  guidelines  for  conducting  feed¬ 
back  data  collection  programs  using  the  structured  interview  method  assessed. 
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GUIDELINES  FOR  CONDUCTING  A  SCHOOL  FEEDBACK  DATA  COLLECTION  PROGRAM 


Figure  A-l  shows  the  steps  involved  in  (1)  planning  and  conducting  a 
feedback  data  collection  program  using  structured  interviews  and  (2)  sum¬ 
marizing  and  using  the  data  for  curriculum  evaluation.  These  steps  are  dis¬ 
cussed  in  detail  in  the  subsequent  paragraphs. 

1.0  PLAN  DATA  COLLECTION  AND  UTILIZATION.  Command  level  "planning"  to 

implement  and  conduct  an  effective  training  appraisal  program  using  the 

recommended  structured  interview  procedure  should  be  directed  at: 

•  assignment  of  responsibilities  to  project  personnel  and  delegation 
of  commensurate  authority 

•  overall  cognizance  of  data  collection/analysis  activities 

•  monitoring  utilization  of  the  data  to  insure  appropriate  course 
revisions  based  on  the  data 

•  followup  on  course  revision  recommendations  when  approval  is  required 
at  higher  level. 

Staff  functions  required  concern  project  coordination,  development  of  data 
collection  instruments,  student  screening/control,  interviewing,  data  analysis, 
and  curriculum  revision.  Each  of  these  functions  are  discussed  below. 

1.1  DESIGNATE  PROJECT  COORDINATOR.  The  structured  interview  method  was 
designed  to  be  straightforward  but  sufficiently  flexible  for  adapta¬ 
tion  to  diverse  training  commands.  The  positions  of  authority,  as 
well  as  the  technical  skills,  most  useful  for  adapting  the  method 

to  a  specific  comnand  may  be  associated  with  personnel  in  several 
departments  or  divisions.  A  relatively  senior  individual  should  be 
designated  as  command  coordinator.  Appropriate  authority  for 
access  to  and  use  of  necessary  personnel  should  be  delegated  to 
that  individual.  The  CIS  officer  is  recommended  for  project  coor¬ 
dination  because:  (1)  the  development  and  use  of  training  appraisal 
procedures  are  already  a  responsibility  of  this  office  and  (2)  CIS 
officers  normally  have  well  established  interdepartmental  relation¬ 
ships.  The  coordinator 's  interest/need  for  the  feedback  data  will 
facilitate  close,  continuing  coordination  of  other  functions 
described  below. 

1.2  DESIGNATE  INSTRUMENT  DEVELOPMENT  PERSONNEL.  The  procedures  recommended 
do  not  require  personnel  thoroughly  trained  in  instrument  develop¬ 
ment  or  data  analysis.  The  most  difficult  job  will  be  to  develop 
suitable  task  statements  for  use  on  the  survey  instruments.  It  is 
recommended  that  individuals  who  are  thoroughly  familiar  with  the 
course  be  tasked  to  develop  the  necessary  job  task  statements.  It  is 
also  recommended  that  the  same  personnel  who  develop  the  instruments 
also  be  tasked  to  accomplish  the  data  analysis.  This  is  suggested  in 
order  to  achieve  a  match  between  instrument  design/data  gathered  and 
the  devised  analysis  scheme. 


37 


TAEG  Report  No.  92 


Figure  A- 


PLAN  DATA  COLLECTION  AND  UTILIZATION 


Major  Steps  Involved  in  Planning,  Conducting,  Analyzing, 
and  Using  Training  Feedback 
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Based  on  the  experiences  gained  in  the  prototype  evaluation,  school 
staff  personnel  tasked  with  curriculum  writing/development  functioning 
under  the  general  supervision  of  command  CIS  personnel  are  recommended. 

1.3  DESIGNATE  STUDENT  SCREENING/CONTROL  RESPONSIBILITIES.  Steps  should 
be  taken  to  insure  that  all  possible  interview  candidates  are 
routinely  screened  and,  if  qualified,  interviewed.  It  is  recom¬ 
mended  that  this  function  be  assigned  to  personnel  who  are  in 
charge  of  the  advanced  students  either  during  check-in  or  after 
convening  of  their  class.  Previous  experience  has  shown  that  this 
is  usually  the  director  of  the  school  offering  the  course,  and  that 
the  director  is  best  represented  for  this  purpose  by  a  selected 
course  instructor  or  learning  supervisor. 

1.4  DESIGNATE  INTERVIEWER (S) .  Senior  military  instructors  who  are  from 
the  same  rating(s)  as  the  individuals  being  interviewed,  and  who 
are  also  assigned  to  the  school  staff,  are  recommended  for  the 
interviewing  task.  Based  on  experience  gained  in  the  study  program, 
senior  petty  officer  instructors  were  enthusiastic  about  improving 
both  their  school's  courses  and  the  expertise  of  sailors  in  their 
own  rating.  They  were  particularly  effective  interviewers  in 
terms  of  the  number  and  specificity  of  amplifying  remarks  collected. 

1.5  DESIGNATE  DATA  ANALYSIS  PERSONNEL.  As  indicated  in  1.2  above, 
it  is  recommended  that  the  same  personnel  who  develop  the  data 
collection  instruments  also  analyze  the  data  collected. 

1.6  DESIGNATE  CURRICULUM  REV  I  EWER ( S ) .  Generally,  curriculum  review  is 
an  ongoing  function  at  every  training  command  and  further  selection 
of  personnel  is  unnecessary.  Those  individuals  who  have  this 
responsibility  should  be  made  aware  of  the  feedback  project  and  the 
availability  of  the  data  for  course  review. 

2.0  DEVELOP  DATA  COLLECTION  INSTRUMENTS.  Instrument  development  requires 
selecting  job  tasks  for  whicn  training  will  be  evaluated  and  designing 
interview  forms  and  associated  procedures  for  their  use.  Interviewee 
background  data  forms  should  also  be  developed  to  collect  information 
concerning  the  experience  of  the  interviewees.  Reason  Codes  are  also 
helpful  for  data  analysis  and  may  be  developed  for  use  as  a  fourth 
"instrument."  These  items  are  discussed  in  more  detail  below. 

2.1  SELECT  JOB  TASKS  FOR  TRAINING  EVALUATION.  Job  tasks  for  evaluation 
may  be  selected  from  existing  job  task  lists  or  may  be  derived  from 
the  learning  objectives  of  the  course.  The  items  on  the  NAVEDTRACOM 
Training  Appraisal  System  (TAS)  Level  II  questionnaires  comprise  one 
source  of  job  task  statements.  Two  other  possible  sources  of  job  task 
lists  a^e  discussed  below.  In  addition,  guidance  is  offered  (see 
attachment  1)  concerning  the  screening  of  established  job  task  lists 
and/cr  development  of  new  job  cask  statements  should  that  be  necessary/ 
desirable. 


39 


TAEG  Report  No.  92 


The  TAS  Level  II  questionnaires,  NOTAP  Job  Task  Inventory,  and 
course  curriculum  comprise  the  three  main  options  for  selection 
of  job  task  statements.  These  are  discussed  below: 

1.  TAS  Level  II  Questionnaires.  The  job  task  statements  on  a 
given  TAS  Level  II  questionnaire  reflect  what  a  student  is 
taught  in  a  particular  course.  These  questionnaires  will 
already  be  familiar  to  many  fleet  personnel  who  have  been 
involved  in  TAS  Level  II  evaluations.  If  available,  the  job 
task  statements  used  on  these  questionnaires  are  recommended 
as  the  basis  for  job  task  statements  to  be  used  on  the  inter¬ 
view  instruments. 

2.  NOTAP  Job  Task  Inventory  (JTI).  The  latest  NOTAP  JTI  for  the 
rating(s)  (and  rate,  as  appropriate)  also  contains  information 
about  the  actual  job  tasks  required  of  recent  school  graduates 
in  the  fleet.  Job  task  statements  for  use  on  the  interview 
form  may  be  extracted  from  these  lists.  It  must  be  recognized, 
however,  that  JTIs  are  lengthy  and  contain  all  types  of  work 
performance  (e.g.,  administrative,  military)  in  addition  to 
technical  training  related  job  tasks.  These  lists  will  require 
an  initial  screening  to  select  technical  tasks  of  the  specialty. 
A  second  screening  may  be  required  to  eliminate  technical 
tasks  performed  by  less  than  20  percent  of  recent  school 
graduates.  This  is  recommended  to  avoid  producing  an  instrument 
which  would  lead  to  an  unacceptably  long  interview.  Note  also 
that  the  JTIs  reflect  the  job  performance  of  personnel  in  each 
rating  by  rate.  It  is  thus  necessary  to  insure  that  the  JTI 
for  the  (approximate)  rates  of  recent  graduates  is  selected. 

3.  Learning  Objectives.  The  learning  objectives  of  the  curriculum 
being  evaluated  comprise  one  other  source  of  possible  job  task 
statements.  Use  of  this  source,  however,  requires  that  con¬ 
siderable  effort  be  expended  to  write  job  task  statements. 

For  example,  shortening  of  learning  objectives,  translating 
or  converting  theory  oriented  learning  objectives  to  corre¬ 
sponding  job  task  statements,  and  screening  for  learning 
objectives  for  nonoperational  equipment  used  only  for  training 
purposes  are  frequently  required.  For  these  reasons,  the 
formulation  of  new  job  task  lists  from  course  learning  objec¬ 
tives  is  recommended  only  when  both  the  TAS  Level  II  question¬ 
naire  and  NOTAP  JTI  are  judged  unacceptable. 

2.2  DESIGN  INTERVIEW  FORMS/PROCEDURES.  Design  of  the  interview  forms 
involves  devising  a  specific  data  collection  format,  instructions 
to  the  interviewees,  and  general  instructions  to  the  interviewers 
concerning  interview  procedures.  These  matters  are  discussed 
below. 
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2.2.1  DEVISE  SPECIFIC  FORMAT.  A  recommended  survey  form  is 
contained  in  the  Interview  Kit  as  attachment  2.  It  is 
similar  to  the  one  used  in  the  TAEG  study  program.  It 
should  be  noted  that  the  three  categories  of  feedback 
solicited  about  graduates'  job  task  performance  (i.e..  Do 
With  Ease,  Do  With  Difficulty,  or  Don't  Do)  refer  to  the 
job  behavior  of  recent  graduates  under  general  supervision. 
The  recommended  instructions  to  the  interviewee  (see  page  73) 
reflect  this  basic  concept.  Three  alternate  sets  of 
choices  for  categorizing  graduate  performance  are  contained 
in  table  A-l .  Other  sets  of  categories  can  be  devised  to 
meet  specific  information  needs  and  desired  data  analysis 
schemes.  It  should  be  noted,  however,  that  the  following 
sections  of  this  report  dealing  with  data  analysis  reflect 
the  use  of  the  recommended  categories  only. 


TABLE  A-l.  ALTERNATE  SETS  OF  GRADUATE  PERFORMANCE  DESCRIBERS 
WITH  SOURCE/PRIOR  APPLICATION 


Option 

Graduate  Performance  Describers 

Source/Prior  Application 

1 

Don ' t  Do 

Do  With  Difficulty 

Do  Adequately  With  Close 
Supervision 

Do  With  Ease 

Suggestions  from  some 
interviewers  and  inter¬ 
viewees  in  the  study 
program 

2 

Training  Less  Than  Adequate 
Adequate 

More  Than  Adequate 

Task  Not  Observed 

CNET  Training  Appraisal 
System 

3 

Task  Not  Observed 

Can  Not  Do 

Can  Do  With  Supervision 

Can  Do  Without  Close 

Supervision 

TAEG's  informal 
experience  in  identifying 
state  of  training  in 
selected  Naval  Reserve 

Units 

2.2.2  DEVELOP  INSTRUCTIONS  TO  INTERVIEWERS.  This  step  involves 
developing  specific  instructions  for  organizing  and 
completing  each  interview  in  order  to  insure  standard¬ 
ization  of  procedure  and  correct  use  of  the  interview 
form.  Attachment  2  contains  recommended  "Interview 
Instructions"  which  are  patterned  after  those  used 
successfully  in  the  study  program.  These  instructions 
can  be  amended  as  necessary  to  meet  individual  command 
needs. 
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2.3  DEVELOP  BACKGROUND  DATA  FORM.  The  Background  Data  Form  should  elicit 
enough  information  about  the  advanced  students  to  determine  how  well 
they  represent  the  larger  fleet  group.  Relevant  questions  might 
concern  the  advanced  students'  most  recent  tour  (i.e.,  whether  or  not 
it  was  sea  or  shore,  type  ship/squadron,  and  type  of  equipment  he  or 
she  worked  on  or  operated).  The  Background  Data  Form  may  also  be  used 
to  seek  answers  to  questions  that  may  be  useful  to  the  training  command 
but  that  may  not  pertain  to  a  specific  job  task.  This  information 
might  relate  to  technical  training  or  military  training,  in  general. 

It  is  recommended,  however,  that  the  Background  Data  Form  be  limited 
to  one  page  in  order  to  facilitate  its  use  in  a  classroom  as  a  screening 
instrument  for  selecting  the  particular  advanced  students  who  will  be 
interviewed.  A  key  question  that  should  appear  on  any  background  data 
form  should  be  similar  to  the  following:  "In  the  last  12  months,  have 

you  had  an  opportunity  to  observe  and  evaluate  recent  _  school 

graduates  in  operational  assignments?"  Followup  questions  that  should 
be  asked  include,  "If  you  answered  'Yes'  to  the  previous  question,  how 
many  people  did  you  observe  and  evaluate?  For  how  long?"  These 
questions  are  essential  to  establish  whether  or  not  an  advanced  student 
has,  in  fact,  observed,  in  an  operational  assignment,  the  job  performance 
of  recent  graduates  of  the  course  being  evaluated.  Attachment  2 
contains  a  sample  Background  Data  Form.  Additional  questions  can/should 
be  added  as  needed. 

2.4  DEVELOP  REASON  CODE  FORM.  Reason  Codes  are  developed  to  reflect 
possible  explanations  concerning  why  recent  graduates  might  have 
had  performance  difficulties  or  might  not  have  performed  a  particu¬ 
lar  task  at  all.  Reason  Codes  will  tend  to  be  largely  rating 
specific;  however,  there  are  some  Reason  Codes  that  have  meaning 

in  many  different  ratings.  Attachment  2  contains  a  list  of  Reason 
Codes  that  may  be  applicable  to  many  ratings.  Reason  Codes  are 
provided  for  both  the  "Don't  Do"  category  and  the  "Do  With 
Difficulty"  category.  Reason  Code  forms  may/should  reflect  the 
Reason  Codes  from  this  list  that  are  deemed  helpful  in  categorizing 
training  appraisal  feedback,  as  well  as  other  locally  developed 
Reason  Codes. 

3.0  ARRANGE  INTERVIEW  FACILITIES.  It  is  important  that  at  least  part  B  of 
the  Interview  Form  be  completed  privately  by  the  interviewer  and 
interviewee.  If  possible,  a  private  office  should  be  reserved  for  the 
interview.  This  provision  may  affect  the  attitude  and  motivation  of 
the  person  being  interviewed  concerning  the  importance  of  the  task 
and,  consequently,  the  degree  of  "effort"  he  will  put  into  his  contri¬ 
bution.  A  nicely  furnished  office  in  a  prominent  part  of  the  building 
will  aid  in  giving  the  impression  that  what  the  advanced  student  is 
doing  is  very  important  and  that  his  full  cooperation/contribution  is 
vital . 

4.0  REVIEW  BASIC  INTERVIEW  METHODS.  The  productivity  of  training  appraisal 
interviews  is  generally  aided  by  interviewer  preparation  and  practice. 
Invariably,  the  selected  interviewers  will  have  had  experience  with 
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some  type  of  interview  situation.  Nevertheless,  review  of  attachment 
3  is  recommended  to  refresh  and  focus  interview er  skills  on  the  training 
appraisal  task.  In  addition,  practice  interviews  using  draft  data 
collection  instruments  will  provide  experience  with  the  specific 
setting  and  population  concerned.  Attachment  3  contains  a  concise 
review  of  military  training  appraisal  interview  procedures.  Several 
other  salient  points  about  interviewing  are  addressed  here.  The 
interviewer  must  accept  that  the  respondent  is  the  best  "expert" 
concerning  his/her  own  ideas  and  opinions  and  that  the  object  of  the 
interview  is  to  learn  about  the  respondent's  ideas  and  opinions.  As 
much  as  possible,  these  ideas  and  opinions  should  be  captured  in  the 
respondent's  own  words.  Sometimes  the  interviewee's  first  response  is 
general  in  nature,  and  it  may  be  necessary  to  ask  probing  questions  to 
get  at  specific  skills  which  are  performed  less  adequately.  Settling 
for  just  the  name  of  a  topic  that  "needs  more  emphasis"  provides 
little  help  to  curriculum  writers.  In  this  situation,  it  may  be 
necessary  to  ask  specific  questions  such  as,  "Do  they  perform  this 
task  too  slowly?"  in  order  to  get  at  the  actual  skill(s)  that  needs  to 
be  taught  differently  or  mere  thoroughly.  On  the  other  hand,  when  the 
initial  response  is  wordy  and  contains  a  great  deal  of  information,  it 
may  be  necessary  to  summarize  and  clarify,  trying  to  restate  the 
essence  of  the  statement  in  a  more  succinct  and  usable  form.  In  doing 
so,  however,  it  is  important  to  avoid  questions  that  bring  up  issues 
that  were  not  raised  by  the  respondent.  The  interviewer  should  not  do 
anything  to  prejudice  the  respondent's  remarks  such  as  arguing  about 
the  accuracy  of  the  data  or  asking  slanted  questions  to  see  if  the 
respondent  has  the  same  opinion  as  the  interviewer  about  some  particular 
issue.  It  is  often  helpful  to  read  back  to  the  respondent  what  has 
been  written  down  to  test  the  accuracy  of  impressions.  Phrases, 
words,  and  sentences  that  tend  to  occur  more  frequently  in  productive 
interviews  are: 

I  understand  you  to  say... 

Do  you  mean  that. . . 

In  other  words . . . 

Could  you  tell  me  specifically  what  behavior  the  graduate  does  not 

perform  adequately... 

Let  me  see  if  I  understand... 

I'm  confused  about  that  point;  could  you  restate  it  for  me... 

Hmmrnm. . . I  see. . . 

Let  me  think  about  this  for  a  minute... 

Tell  me  more  about  that... 

Words,  phrases,  and  sentences  that  tend  to  arise  in  unproductive  interviews 
are: 


I  don't  understand  you... 

Do  you  really  think  that... 

I  disagree. . . 

Don't  you  think  that... 

That's  confusing;  say  it  again... 

What  you  really  mean  to  say  is... 

Did  you  have  the  same  problem  I  did  with  getting  them  to  do... 
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It  is  vital  to  remember  that  the  respondent  is  the  expert  on  his/her 
own  ideas  and  opinions.  The  interviewer's  job  Ti"~to  aid  in  the  expression 
of  his/her  ideas  and,  thus,  obtain  specific  information  concerning  the 
need  for  and  nature  of  possible  curriculum  changes.  This  means  obtaining 
and  recording  information  about  skills  and  knowledge  which  at  some 
point  can  be  associated  with  a  specific  terminal  or  enabling  objective 
of  a  course's  curriculum. 

5.0  SELECT  INTERVIEW  POINT  IN  ADVANCED  STUDENTS'  CHECK-IN/TRAINING  SCHEDULE. 
Early  contact  with  the  NAVEDTRACOM  advanced  students  is  recommended  to 
maintain  their  credibility  as  fleet  representatives.  Scheduling  the 
interview  during  the  check  in  process,  or  during  the  period  prior  to 
or  shortly  after  the  course  convening  date,  will  generally  meet  this 
requirement. 

6.0  PREPLAN  INTERVIEWS.  It  is  necessary  to  determine  which  NAVEDTRACOM 
advanced  students  are  "qualified1'  to  provide  information  about  the 
adequacy  of  a  school's  training  for  the  fleet.  Criteria  should  be 
established  concerning  the  degree  of  opportunity  to  observe  recent 
school  graduates  on  the  job  in  operational  assignments  and  the  recency 
and  length  of  this  observation.  All  "qualified"  individuals  should  be 
scheduled  for  interviews.  In  addition,  a  purposeful  attempt  should  be 
made  to  encourage  interviewee  cooperation  and  enhance  his/her  motiva¬ 
tion  for  the  interview. 

6.1  SCREEN  INTERVIEWEES.  The  Background  Data  Form  (see  attachment  2) 
is  recommended  for  use  in  student  screening.  A  "yes"  answer  to 
the  question  concerning  opportunity  to  observe  recent  school 
graduate  performance  on  the  job  is  a  prerequisite  for  considera¬ 
tion.  Ensuring  that  all  advanced  students  in  the  designated 
rating(s)  are  screened  will  result  in  the  maximum  number  of 
individuals  being  interviewed  and  the  most  data  available  collected. 

6.2  SELECT  INTERVIEWEES.  Meeting  established  criteria  for  numbers 
of  recent  graduates  observed,  and  duration  of  observation,  is 
necessary  for  inclusion  of  a  student  in  the  interview  sample.  In 
the  TAEG  study  program,  all  individuals  were  selected  for  inter¬ 
view  who  had  observed  at  least  one  recent  graduate  for  at  least  3 
months.  Individuals  from  the  U.S.  Marine  Corps  and  other  services 
who  met  this  criterion  were  also  included.  No  further  criteria 
were  established  concerning,  for  example,  USN  versus  USNR,  male 
versus  female,  or  senior  versus  junior  petty  officer. 

6.3  SCHEDULE  INTERVIEWS.  Individuals  identified  as  "qualified" 
should  be  scheduled  for  their  interview  to  occur  as  soon  as 
possible  after  their  arrival  at  the  school.  This  is  considered 
necessary  to  avoid  continued  exposure  to  training  command  "problems" 
which  may  bias  their  responses. 

6.4  INSURE  DESIGNATED  INTERVIEWEES  ARE  INTERVIEWED.  A  simple  system  to 
maintain  accountability  for  reporting  will  help  prevent  possible 
lost  interviews/data.  Many  ratings  have  a  low  throughput  of 
advanced  students  and  the  loss  of  eligible  interviewees  can 
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result  in  a  considerable  decrease  in  data.  Conversely,  in  ratings 
where  the  throughput  is  high,  a  sample  of  all  eligible  interviewees 
might  provide  sufficient  data. 

6.5  ENCOURAGE  SELECTED  INTERVIEWEES  TO  CONTRIBUTE  AS  MUCH  AS  POSSIBLE. 
Most  sailors,  regardless  of  their  current  attitude  about  Naval 
service,  are  usually  deeply  proud  of  their  particular  work  and 
rating  and  tend  to  identify  strongly  with  the  community  of  indi¬ 
viduals  in  that  rating.  This  can  be  a  strong  help  in  soliciting 
their  cooperation  and  help  in  getting  information  that  will 
improve  training  for  their  rating. 

7.0  INTERVIEW  DESIGNATED  STUDENTS.  Detailed  procedures  for  conducting 
interviews  are  contained  in  attachments  2  and  3.  Other  suggestions 
were  provided  under  several  of  the  preceding  steps.  Familiarity  with 
these  materials  and  completion  of  practice  interviews  should  make 
conduct  of  the  actual  interviews  straightforward.  Consequently,  no 
further  guidance  in  this  area  is  provided  in  the  body  of  the  text. 

Once  interviewing  begins,  however,  continuing  effort  is  necessary  to 
insure  that  the  maximum  amount  of  useful  data  is  collected.  The 
effort  required  is  described  below. 

7.1  MAINTAIN  OPEN  COMMUNICATIONS  WITH  SCREENING  PERSONNEL.  Open 
communications  with  screening  personnel  should  be  maintained  to 
preclude  lost  data  due  to  reasons  such  as  scheduling  confusion  or 
undesirable  time  constraints. 

7.2  TRANSMIT  INTERVIEW  FORMS  TO  DATA  ANALYSIS  PERSONNEL.  It  is 
recommended  that  data  be  processed  on  an  "as-collected''  basis. 

This  will  allow  routine  inspection  of  the  data  to  determine 
possible  emerging  training  problems  and  to  identify  specific 
areas  for  further  detailed  inquiry  during  subsequent  interviews. 

8.0  MANAGE/SUMMARIZE/ANALYZE  DATA.  Obtaining  maximum  benefit  from  the 
feedback  data  collection  program  requires  proper  management,  summari¬ 
zation,  and  analysis  of  all  data  collected.  Recommended  procedures 
for  these  efforts  are  provided  below. 

8.1  MANAGE  DATA.  Management  of  the  data  involves  accurate  and  timely 
compiling  of  information  collected,  as  well  as  monitoring  the 
degree  of  fleet  representativeness  of  the  interviewees  and  the 
quality  of  interview  data  being  collected. 

8.1.1  COMPILE  DATA.  Interviewee  Background  Data  as  well  as 
actual  interview  data  must  be  recorded. 

8. 1.1.1  MAINTAIN  FILE  OF  ALL  COMPLETED  BACKGROUND 
DATA  FORMS.  It  is  recommended  that  a  file  of  Back¬ 
ground  Data  Forms  for  all  individuals  screened  be 
retained  at  the  school.  This  information  is  of  immediate 
concern  in  determining  the  percentage  of  advanced 
students  in  the  selected  ratings  actually  eligible  for 
interview  and  in  determining  the  fleet  representativeness 
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of  the  group  of  advanced  students  interviewed.  The 
background  information  might  also  be  useful  to  advanced 
course  managers  in  determining  student  profiles  and 
assessing  advanced  course  curriculum  needs. 

8. 1.1. 2  RECORD  INTERVIEW  DATA.  Careful  recording  of 

the  interview  data  is  required  to  maintain 
integrity  of  the  data  base  and  to  permit 
ready  summarization  for  subsequent  analysis. 

A  Data  Collection  Worksheet  form  that  can  be 
used  to  record  interview  data  about  each  job 
task  listed  on  the  Interview  Form  is  contained 
in  figure  A-2.  The  number  of  the  task  (from 
the  Interview  Form)  and  its  name  should  be 
written  across  the  top.  Space  is  allocated 
on  the  worksheet  to  tabulate  the  number  of 
times  the  interviewees  classified  performance 
of  the  task  as  "Do  With  Ease,"  "Do  With 
Difficulty,"  or  "Don't  Do."  In  addition, 
space  is  provided  to  tabulate  Reason  Code 
selections  for  tasks  judged  to  be  in  the 
"Don't  Do"  or  "Do  With  Difficulty"  categories. 
Finally,  all  comments  made  by  interviewees 
pertaining  to  each  task— either  to  clarify 
why  particular  Reason  Codes  were  selected  or 
to  add  amplifying  information— can  be  recorded 
on  the  back  along  with  the  Reason  Codes  that 
accompanied  the  remarks.  Comments  received 
in  response  to  the  open-ended  question  at  the 
end  of  the  Interview  Form  should  probably  be 
compiled  separately  on  blank  sheets  of  paper 
so  that  those  statements  will  not  be  confused 
with  specific  statements  made  in  response  to 
a  given  job  task.  Also  in  the  interview  the 
importance  attached  to  more  spontaneous 
general  comments  may  be  different;  they  may 
require  separate  treatment/consideration. 

8.1.2  MONITOR  DATA  BASE.  All  interview  data  received 
should  be  routinely  reviewed  to  identify  nonpro¬ 
ductive  interview  techniques  which  may  require 
remediation  (e.g.,  unclear  or  nonspecific  comments) 
In  addition,  even  cursory  review  of  the  data  may 
also  suggest  trends  in  graduate  performance  informa 
tion  that  may  not  be  apparent  to  individual  inter¬ 
viewers.  Alerting  interviewers  about  such  trends 
may  facilitate  collection  of  more  useful  data  in 
that  area. 

8.1.3  DETERMINE  REPRESENTATIVENESS  OF  INTERVIEW  SAMPLE. 
The  background  data  on  interviewees  should  be 
examined  to  determine  how  well  the  interviewed 
group  represents  the  larger  fleet  group.  This 
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TASK  (Name  and  Interview  Form  number): 

\ 

Number  "Do  With  Ease"  Selections: 

Number  "Don't  Do"  Choices  and  "Don't  Do" 

Reason 

Code  Selections: 

Don't  Do  Choices: 

Reason 

Codes : 

A 

B 

C 

D 

Number  "Do  With  Difficulty"  Choices  and  ' 
Selections: 

’Do  With 

Difficulty"  Reason  Codes 

Do  With  Difficulty  Choices: 

Reason  Codes 

E 

Q 

CC 

F 

R 

DD 

G 

S 

EE 

H 

T 

FF 

I 

U 

GG 

J 

V 

HH 

K 

W 

II 

L 

X 

JJ 

M 

Y 

KK 

N 

Z 

LL 

0 

AA 

MM 

P 

BB 

NN 

Figure  A-2.  Recommended  Data  Collection  Worksheet 
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REASON  CODES  WITH  CORRESPONDING  COMMENTS: 

Reason  Code(s)  Comments 


Figure  A-2.  Recommended  Data  Collection  Worksheet  (continued) 
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SUMMARIZt  DA’A.  ate  summari zati on  involves  prima-  ,  .alcula*’ou 
the  percentage  of  interviewee  response  choices  ’o  >everal  rev 
categories  which  are  important  to  various  data  analysis  method1. 

The  data  analvsi.,  methods  are  discussed  in  a  subsequent  secrior 
of  this  appendix,  ''r  itedures  for  calculating  selec  ted  percentage' 
are  explained  and  demonstrated  below. 

8.2.1  CALCULATE  GRADUATE  "Til  !  ,’AT  EON  PERCENTAGE  r  .  The  qraduate 

utiliZftior  percentage  ts  the  degree  to  which 

recent  gracuates  are  used  tc  perform  the  iob  tasks  at 

the  work  renters  represented  by  the  interviewees. 

"he  utiliza  ion  percentage  r  .  alcul  ate::  for  each  job 
task  by  dividing  the  sum  of  all  "Do  With  Ease  and  Ho 
With  Diffir.lty"  responses  by  the  total  number  of 
responses  and  multiplying  by  IDO.  ‘‘inure  A- 3  contains 
a  sample  computation.  For  between  task  comparison 
purposes  the  iob  task  statements  may  be  listed/ranked 
on  the  basis  of  the  utilization  percentage. 

8.2.2  CALCULATE  TRAINING  PROFICIENCY  PERCENTAGES.  The  traininq 
proficiency  percentage  reflects  the  deqree  to  which  recent 
graduates,  under  routine  supervision,  are  performing  their 
assigned  tasxs  with  ease.  Thiu  should  be  calculated  for 
each  task  in  order  to  make  judaments  about  instructional 
effectiveness.  The  Data  Collection  Worksheet  information 
is  used  for  this.  The  worksheet  information  also  helps 
sort  out  those  performance  difficulties  not  attributable 
to  training.  The  training  proficiency  percentage  is 
calculated  for  each  task  by  dividing  the  "corrected" 
number  of  "Do  With  Ease"  selections  by  the  sum  of  all 

"Do  With  Ease"  and  "Do  With  Difficulty"  responses  and 
multiplying  the  quotient  by  100.  The  "corrected" 
number  of  "Do  With  Ease"  selections  is  calculated  by 
adding  to  the  actual  number  of  "Do  With  Ease"  selections 
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Data  Collection  Worksheet  information  for  job  task  statement  number  14  (Tie 
nautical  knots): 

Do  With  Case:  15 

Do  With  Difficulty:  13 

Don't  Do:  8 

Total  Responses:  36 

Utilization  Percent  =  (Number  Do  With  Ease  +  Number  Do  With  Difficulty)  x  100 

Number  Total  Responses 


Uti 1 ization 
Uti 1 ization 


Percent  =  (15_^13)  x  100 
Percent  =  (.778)  x  100  =  77.8 

Figure  A-3.  Sample  Utilization  Percentage  Computation 


Data  Collection  Worksheet  information  for  job  task  statement  number  14  (Tie 
nautical  knots): 

Do  With  Ease:  15 


Do  With  Difficulty:  13 

Do  With  Difficulty  Selections  Determined  "Not  Training  Related":  3 
Don ' t  Do :  8 


Total  Responses:  36 
Training  Proficiency  Percent  - 


_ Corrected  Number  Do  With  Ease _ inn 

(Number  Do  With  Ease  +  Number  Do  With  Difficulty)  x 

(Corrected  Number  Do  With  Ease  =  Number  Do  With  Ease  +  Number  Do  With 
Difficulty  Selection  Determined  "Not  Training  Related") 


Training  Proficiency  Percent  = 


IS 

rnr+~‘T3T 


x  100 


Training  Proficiency  Percent  -  (.643)  x  100  =  64.3 


Figure  A-4.  Sample  Training  Proficiency  Percentage  Computation 
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direct  use  of  the  utilization  percentages  and  training  proficiency 
percentages  to  make  respective  judgments  about  course  relevancy 
and  training  effectiveness.  A  second  method  features  careful 
review,  by  subject  matter  experts,  of  each  interviewee  comment, 
each  task's  Reason  Code  frequency  distribution,  and  the  overall 
Reason  Code  selection  percentages.  The  third  method  involves  the 
use  of  an  experimental  matrix  method  to  display  graphically  the 
relationship  of  training  adequacy  data  to  task  importance  data. 

These  three  data  analysis  methods  are  described  below. 

8.3.1  USE  UTILIZATION  AND  TRAINING  PROFICIENCY  PERCENTAGES  TO 
ASSESS  RELEVANCY  OF  COURSE  CURRICULUM  AND  ADEQUACY  OF 
TRAINING.  The  utilization  percentage  indicates  the 
degree  to  which  recent  graduates  are  actually  employed 
at  the  tasks  for  which  they  received  training.  This 
bears  on  the  relevancy  of  the  course.  In  a  similar 
manner,  training  proficiency  percentages  indicate  the 
degree  of  ease  with  which  graduates  perform  the  tasks 
for  which  they  were  trained.  This  is  related  to  the 
training  adequacy  of  the  course.  Both  course  relevancy 
and  training  adequacy  are  discussed  in  this  section. 

8. 3. 1.1  DETERMINE  COURSE  RELEVANCY.  Specific 

criteria  for  making  absolute  judgments  about 
course  relevancy  from  data  collected  with  this 
method  do  not  exist;  consequently,  experimental 
criteria  for  assessing  the  relevancy  of  training 
at  individual  tasks  were  arbitrarily  established 
and  used  in  the  prototype  study.  Judgments 
about  course  relevancy  can  then  be  made  based 
on  a  review  of  training  relevancy  data  for 
individual  job  tasks.  Criteria  for  high, 
moderate,  and  low  relevancy  for  individual 
task  training,  using  utilization  percentages, 
are  contained  in  table  A-2. 


TABLE  A-2.  EXPERIMENTAL 

RELEVANCY  CRITERIA 

Task  Utilization  Percentage 

Experimental  Relevancy  Criteria 

75  -  100 

High 

50  -  74 

Moderate 

0  -  49 

Low 
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The  assignment  of  these  experimental  criteria 
is  based  on  the  following: 

HIGH 

A  high  utilization  value  could/should  be 
assigned  if  three- fourths  or  more  of  the 
respondents  report  that  recent  graduates  are 
employed  at  a  particular  task.  It  is  assumed 
that  it  is  highly  necessary  that  appropriate 
training  be  provided  in  the  course. 

LOW 

A  low  utilization  value  could/should  be 
assigned  if  less  than  half  of  the  respondents 
indicated  "their11  graduates  are  employed  at 
the  task.  It  is  assumed  that  these  tasks  are, 
on  the  whole,  the  least  critical  for  formal 
training  and  that  with  limited  time  and  personnel 
resources  they  could  be  considered  as  candidates 
for  a  reduction  of  training  attention. 

MODERATE 

A  moderate  utilization  value  was  assigned 
arbitrarily  for  any  utilization  percent  between 
"high"  and  "low." 

Good  judgment  is  required  in  using  utilization 
percentage  data  as  a  basis  for  course  change. 

Some  job  tasks  may,  in  fact,  be  infrequently 
performed  but  permit  little  tolerance  for 
error  or  delay  when  they  must  be  performed 
(e.g.,  life  saving  tasks,  emergency  responses). 
The  experience  and  judgment  of  course  managers 
should  prevail  in  matters  of  this  kind.  The 
prcoosed  arbitrary  experimental  labels  and 
suggested  conclusions  are  offered  as  guidelines 
only. 

DETERMINE  INSTRUCTIONAL  EFFECTIVENESS. 
Instructional  effectiveness  is  related  to,  but 
not  defined  by,  the  degree  of  ease  with  which 
graduates  perform  the  tasks  for  which  they 
were  trained.  The  ease  of  job  task  performance 
is,  however,  a  main  consideration  and  the  data 
collected  from  interviews  can  be  used  to  make 
tentative  judgments  about  course  effectiveness. 

As  with  course  relevancy,  specific  criteria  for 
making  absolute  judgments  about  instructional 
effectiveness  from  data  collected  with  this 
method  do  not  exist.  Consequently,  experimental 
criteria  were  arbitrarily  established  and 


51 


TAEG  Report  No.  92 


utilized  in  the  prototype  study.  "Criteria" 
for  high,  moderate,  and  low  training  effective¬ 
ness  for  individual  task  training,  using 
training  proficiency  percentages,  are  contained 
in  table  A-3. 


TABLE  A-3.  EXPERIMENTAL  EFFECTIVENESS  CRITERIA 


Task  Training 

Proficiency  Percentage 

Experimental 
Effectiveness  Criteria 

75-100 

High 

50-74 

Moderate 

0-49 

Low 

The  assignment  of  the  experimental  criteria  is 
based  on  the  following  considerations: 

HIGH 

A  high  training  adequacy  value  could/should 
be  assigned  if  three-fourths  or  more  of  the 
respondents  report  that  recent  graduates  with 
general  supervision  performed  the  task  with 
ease  (or  did  not  have  training  related  dif¬ 
ficulties).  It  is  assumed  that  there  is  no 
significant  training  problem.  This  "decision" 
was  supported  in  part  by  the  fact  that  the 
SMEs  who  evaluated  usefulness  of  feedback  data 
in  the  TAEG  study  program  generally  reached 
consensus  that  no  performance/training  problem 
existed  if  at  least  70  to  75  percent  of  the 
respondents  reported  their  graduates  performed 
a  task  with  ease. 

LOW 

A  low  training  adequacy  value  could/should  be 
assigned  if  over  half  of  the  respondents 
indicated  "their"  graduates  performed  a  task 
with  difficulty  (and  the  difficulties  were 
training  related).  It  is  assumed  that  training 
is  inadequate  for  tasks  in  this  category. 

MODERATE 

A  moderate  value  is  assigned  arbitrarily  for  any 
task  rated  between  high  and  low. 
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Using  the  same  reasoning  as  with  the  course 
relevancy  analysis,  the  labels  proposed  in 
this  section,  and  conclusions  drawn  from  them, 
should  be  used  as  guidelines  only. 

USE  SUBJECT  MATTER  EXPERTS  TO  PINPOINT  SPECIFIC  CURRICULUM 
STRENGTHS/WEAKNESSES.  Review  of  training  adequacy  data 
by  subject  matter  experts  is  essential  for  determining 
the  specific  strengths  and  weaknesses  of  a  course. 

Subject  matter  experts  can  provide  the  depth  of  under¬ 
standing/interpretation  that  mathematical  analysis 
schemes  can  not  accomplish.  Both  the  tone  and  the 
technique  of  the  SME  review  are  important  for  maximum 
benefit  to  course  evaluation  and  the  instruction  future 
students  will  receive. 

The  tone  of  the  ^jta  analysis  must  be  balanced  and 
professional.  It  mu^t  be  both  reinforcing  of  the  successes 
that  the  training  managers  and  instructors  are  enjoying 
and  objectively  and  professionally  critical  of  areas 
where  course  improvement  is  clearly  required.  Such  a 
balanced  interpretation  of  training  feedback  data  will 
help  prevent  inadvertent  undermining  of  curriculum 
strength  in  an  attempt  to  improve  areas  of  instructional 
inadequacy.  Through  avoidance  of  over  negativism,  it 
will  also  helo  minimize  personal /organi zationai  defensive¬ 
ness  which  can  hamper  training  improvement  by  fostering 
denial  or  rationalization  of  negative  training  feedback 
and/or  feedback  gathering  methods. 

A  positive  and  professional  tone  coupled  with  any 
logical  technique  for  SME  data  analysis  will  probably 
provide  useful  data  for  course  assessment/improvement. 

The  technique  described/recommended  below  is  a  reflection 
of  "lessons  learned"  in  the  prototype  study  taken  "one 
step  beyond."  It  is  a  process  which  should  be  tried  and 
then  modifiec/improved  to  suit  each  command's  unique 
needs.  The  SMEs  must  determine  in  an  overall  way  the 
job  task  areas  that  do  not  pose  any  significant  training 
problem(s)  as  well  as  the  precise  nature  of  the  performance 
problem(s)  actually  uncovered  by  the  data.  Based  on  the 
detailed  appraisal  of  performance  problem  information, 
the  SMEs  must  define  the  training  problem  in  terms  of 
what  students  are  failing  to  learn  and  then  recommend 
where  (i.e.,  in  which  segment(s)  of  the  training  pipeline) 
remediation  is  most  appropriate. 

8. 3. 2.1  DESIGNATE  SMEs.  Approximately  five  SMEs 

should  bp  selected  in  order  to  insure  the  data 
are  interpreted  by  a  broadly  experienced  but 
yet  manageable  group  of  individuals.  The  SMEs 
knowledge  of  the  complete  training  pipeline  is 
also  important.  Ideally,  the  SMEs  should  be 
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drawn  from  the  staff  of  the  school  that  teaches 
the  course  under  evaluation  and  from  representative 
commands  in  other  segments  of  the  training 
pipeline.  This  includes  a  fleet  supervisor 
who  can  represent  the  point  of  view  of  the  on- 
the-job  trainer.  Reviews  of  courses  that  are 
basic  to  a  rating  (e.g.,  "A"  courses)  might 
benefit  from  including  the  complementary 
correspondence  course/rate  training  manual 
writer  in  the  group  of  SMEs. 

Correspondence  courses  and  rate  training 
manuals  may  be  seen  as  one  segment,  or  portion 
of  the  on-the-job  training  segment,  of  the 
training  pipeline. 

8. 3. 2. 2  REVIEW  INTERVIEW  DATA.  The  procedures  selected/ 
recommended  for  SME  review  of  the  data  are 
designed  to  maximize  the  quality/usefulness  of 
their  efforts  both  as  individuals  and  as  a 
group/panel.  They  should  result  in  a  well 
thought  out,  well  discussed,  clear  set  of 
recommendations  to  training  managers. 

The  SMEs  should  be  furnished  a  suirmary  of 
the  feedback  data  for  each  job  task  showing: 

•  frequency  counts  for  all  interviewee 
choices 

•  utilization  percentages 

«  training  proficiency  percentages 

•  all  interviewee  comments. 

In  addition,  each  SME  should  have  the  overall 
Reason  Code  selection  percentages. 

Each  SME  should  evaluate  the  training 
feedback  individually  on  a  task  by  task  basis 
completing  a  worksheet  for  each  job  task 
similar  to  the  sample  in  figure  A-5.  Use  of 
the  worksheet  requires  that  the  SME  complete 
the  following  steps: 

a.  Determine  if  the  data  implies  a 
significant  performance  problem. 

b.  If  no  performance  problem  is  ascertained, 
indicate  which  segments  of  the  training 
pipeline  could  possibly  benefit  from 

the  feedback  and  go  on  to  the  next 
task. 
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1.  PERFORMANCE  ASSESSMENT:  (check  the  appropriate  box;  if  a  or  b  is  selected  complete  only  section  3.) 

/  7  a.  Data  suggest  little  difficulty  in  performance;  therefore  no  training  problem  is  apparent. 

/  7  b.  Data  suggest  difficulty  in  performance,  but  do  not  imply  training  is  the  primary  cause  of 
difficulty. 

/  /  c.  Oata  suggest  difficulty  in  performance  which  may  be  related,  in  part,  to  a  training  problem. 

2.  PROBLEM  DESCRIPTION: 

a.  Performance  problem:  (Define  in  specific  behavioral  terms  what  recent  graduates  are  not  doing 
properly  on  the  job.  Example:  Recent  graduates  log  out  classified  publications  incorrectly, 
leave  sensitive  equipment  in  unlocked  spaces,  and  leave  confidential  MRt  cards  lying  around.) 


b.  Training/learninq  problem:  (Define  in  specific  behavioral  terms  what  students  are  apparently  having 
trouble  learning.  Example:  Students  are  not  remembering  basic  rules  for  insuring  security  to 
classified  equipment  and  documents  {MRC  cards,  in  particular);  they  are  not  learning  to  include 
time  for  proper  handling  of  classified  material  in  their  work  plans;  they  are  not  developing  an 
attitude  of  appreciation  for  proper  security  precautions.) 


3.  RELEVANCE  OF  DATA:  (Check  the  appropriate  box  for  each  segment  of  the  training  pipeline.) 


TRAINING  PIPELINE’ 


Feedback  data  unimpor¬ 
tant  and/or  irrelevant 
to  this  part  of  pipe¬ 
line;  no  action  war¬ 
ranted 


Feedback  data  might  be 
helpful  to  this  part 
of  pipeline  and  data 
should  be  distributed 
thereto 


Feedback  data  moderate¬ 
ly  important  to  this 
part  of  pipeline  and 
should  be  considered 
during  the  next 
regular  review 


Feedback  data  very 
important  and  a  re¬ 
vision  or  modifica¬ 
tion  should  be 
considered  at  this 
time 


Figure  A-5.  Sample  SME  Data  Analysis  Worksheet 
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c.  If  a  performance  problem  is  determined: 

(1)  Define  the  problem  in  specific 
behavioral  terms  (i.e.,  what  is 
the  graduate  doing  improperly 
or  not  doing). 

(2)  Define  the  learning  problem  in 
specific  behavioral  terms 
(i.e.,  what  is  the  graduate  not 
learning  in  the  course). 

(3)  Determine  which  segment(s) 

of  the  training  pipeline  should 
attempt  to  remediate  the  learning 
problem,  and  the  urgency  of 
remediation. 

Following  individual  assessment  of  the 
data,  the  SMEs  should  meet  as  a  group/panel 
and  discuss  their  individual  conclusions.  The 
panel  should  be  instructed  to  achieve  a 
consensus  about  the  definition  of  the  per¬ 
formance  and  learning  problems  as  well  as  the 
recommended  location(s)  in  the  pipeline  for 
remediation  and  the  urgency  of  remediation. 

This  can  be  accomplished  by  completing  the 
same  worksheet  as  a  group/panel  on  a  task  by 
task  basis  as  each  SME  did  individually.  The 
process  of  consensus  usually  assures  careful  dis¬ 
cussion,  exploration  of  differences  of  opinion, 
and,  in  general,  an  in-depth  analysis  of  the 
data.  It  avoids  some  of  the  pitfalls  of 
reliance  on  just  one  senior  SME  who  will 
inevitably  have  his/her  own  biases,  and  it 
avoids  the  often  confusing  display  of  frequency 
distribution  data/votes  that  a  simple  comparison 
of  individual  analyses  might  yield. 

When  SME  consensus  about  the  data  is 
reached,  it  is  clear  which  performance  problems 
have  been  uncovered  by  the  data,  what  the 
corresponding  learning  problems  are,  and  where 
in  the  training  pipeline  remediation  is  most 
appropriate. 

Two  questions  remaining  are  which  of  the 
learning  problems  should  be  addressed  first 
(i.e.,  the  prioritization  problem)  and  how  to 
correct  the  learning  problem  (i.e.,  the  curriculum 
revision  problem).  Solutions  to  the  prioritiza¬ 
tion  problem  were  partially  met  through  the 
panel  discussion  on  the  urgency  of  remediation. 
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The  next  section  explores  an  additional 
technique  for  prioritizing  the  use  of  training 
remediation  resources.  Section  9.0  provides 
experimental  guidelines  for  creating  curriculum 
change. 

8.3.3  DERIVE  EXPERIMENTAL  MATRIX.  The  feedback  information 
obtained  from  the  interviews  may  identify  a  number  of 
job  tasks  as  possibly  "undertrained."  This  identification 
is  based  on  the  single  dimension  (variable)  of  performance 
adequacy  as  reflected  by  the  ease/difficulty  with  which 
graduates  perform  tasks.  In  many  instances,  it  may  not 
be  possible  to  correct,  with  available  resources,  all 
areas  of  "undertraining"  that  are  revealed  by  feedback 
data.  In  such  cases  it  may  be  desirable  to  identify 
those  areas  of  deficiency  most  in  need  of  training 
attention.  Also,  within  a  circumscribed  resource  system, 
it  may  be  desirable  to  identify  training  areas  from 
which  resources  can  be  diverted  to  correct  deficiencies. 

One  technique  that  can  be  used  for  achieving  these  goals 
involves  determining  and  assessing  the  relationship(s) 
between  variables  that  are  equally  important  to  decisions 
about  training.  For  example,  knowledge  of  relationships 
between  how  well  individuals  can  perform  a  task(s)  on 
the  job  and  how  important  it  is  that  they  be  able  to 
perform  it  can  be  used  to  aid  significantly  decisions 
about  how  to  use  training  resources. 

A  matrix  was  developed  by  TAEG  for  assessing  the 
interview  data.  The  method  is  based  on  suggestions  made 
by  Siegel,  Schultz,  and  Federman  (1961).  Determining 
undertrained/overtrained  tasks  involves  relating  ratings 
of  graduates'  adequacy  of  performance  to  estimates  of 
the  relative  importance  of  specific  tasks  to  the  accomplish¬ 
ment  of  the  job.  A  basic  assumption  underlying  use  of 
the  method  is  that  highly  critical  tasks  should  be  trained 
to  a  high  level  of  proficiency  while  less  proficiency 
can  be  tolerated  on  less  critical  tasks.  Tasks  that  are 
performed  poorly  in  relation  to  their  importance  are 
assumed  to  be  undertrained.  Tasks  which  are  performed 
well  on  the  job  but  which  are  relatively  unimportant  are 
assumed  to  be  overtrained. 

Application  of  the  method  requires  the  assignment 
of  performance  adequacy  indexes  (values)  and  importance 
or  "criticality"  indexes  to  the  various  job  tasks. 
Performance  adequacy  indexes  assigned  are  based  on  the 
percent  of  respondents  reporting  that  the  graduates  in 
their  work  centers  either  did  a  task  with  ease  or  had  no 
training  related  difficulties.  The  assumption  is  made 
that  the  respondents'  judgments  are  valid  indications  of 
adequacy  of  graduate  performance  on  the  job.  "Criticality" 
indexes  assigned  are  based  on  degree  of  graduate  utilization 
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at  each  task.  The  assumption  is  made  that  this  is  a 
useful  measure  of  criticality.  It  is  acknowledged  that 
graduate  utilization  to  perform  tasks  may  not  hold  as  a 
valid  indicator  of  criticality  in  all  situations.  Those 
tasks  for  which  there  are  important  consequences  of 
first-performance-failure  (e.g.,  emergencies)  may  be 
just  one  instance  of  overriding  considerations.  Utiliza¬ 
tion  frequency,  as  a  measure  of  criticality,  however, 
still  may  provide  an  important  contribution  to  making 
cost  effective  decisions  about  the  expenditure  of 
training  resources. 

There  are  three  possible  parformance  adequacy 
indices  and  three  corresponding  criticality  indices-- 
"high,"  "moderate,"  and  "low."  The  performance  adequacy 
index  is  based  on  the  training  proficiency  percentage, 
and  the  criticality  index  is  based  on  the  utilization 
percentage  (see  steps  8.2.1  and  8.2.3).  The  indices  are 
related  directly  to  the  experimental  criteria  developed 
for  the  separate  assessments  of  course  relevancy  and 
instructional  effectiveness  and  the  rationale  for  assign¬ 
ment  of  values  is  the  same  (see  tables  A-2  and  A-3). 

Table  A-4  contains  the  basis  for  assignment  of  performance 
adequacy  indices. 


TABLE  A-4.  PERFORMANCE  ADEQUACY  INDICES 


Training  Proficiency  Percentage 

Performance  Adequacy  Index 

75-100 

High 

50-74 

Moderate 

0-49 

Low 

Table  A-5  contains  the  basis  for  assignment  of  criticality  indices. 
TABLE  A-5.  CRITICALITY  INDICES 


Utilization  Percentage  Criticality  Index 


75-100 

50-74 

0-49 
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The  performance  adequacy  and  criticality  indices 
are  determined  for  each  task  listed  in  the  interview 
survey.  The  survey  task  numbers  are  then  listed  in  the 
cell  of  the  matrix  corresponding  to  the  indices  assigned 
to  that  task.  Figure  A-6  contains  the  experimental 
training  assessment  matrix.  Figure  A-7  contains  a 
sample  derivation  of  indices,  and  figure  A-6  contains 
the  correct  plot  of  the  sample  job  task. 

The  cells  of  the  matrix  show  five  possible  assess¬ 
ments  of  training  for  a  specific  job  task: 

•  significant  undertraining 

•  moderate  undertraining 

•  optimal  training 

•  moderate  overtraining 

•  significant  overtraining. 

Tasks  falling  into  the  optimal  training  cells  reflect 
the  principle  that  when  the  performance  adequacy  index 
value  matches  the  criticality  index  value  the  task  has 
been  optimally  trained.  When  the  performance  adequacy 
index  is  higher  than  the  criticality  index,  some  over¬ 
training  may  have  occurred.  When  the  performance  index 
is  lower  than  the  criticality  index,  undertraining  may 
have  occurred. 

Thus,  the  matrix  method  classifies  tasks  as  being 
optimally  trained,  significantly  or  moderately  under¬ 
trained,  and  as  being  significantly  or  moderately  over¬ 
trained.  This  classification  may  assist  prioritization 
of  training  remediation  resources. 

9.0  REVISE  CURRICULUM.  The  Instructional  Systems  Development  (ISD)  model  used 
within  the  NAVEDTRACOM  requires  the  use  of  external  (and  internal)  feed¬ 
back  to  determine  the  nature  of  training  problems.  It  directs  corrective 
action  to  be  applied  at  the  particular  point  in  the  systematic  process 
that  provides  the  technical  procedures  required  to  solve  the  problem. 
Curriculum  change  for  a  specific  course  is  a  highly  individualized  process 
that  is  a  function,  in  part,  of  command  prerogative,  technical  expertise, 
professional  creativity,  physical  facilities,  funding,  complex  organiza¬ 
tional  support  hierarchies,  and  a  variety  of  other  unpredictable  influ¬ 
ences.  It  is  thus  impossible  to  specify  precise  universal  procedures  for 
utilizing  feedback  data;  however,  fundamental  guidance  through  the 
methodology  of  the  ISD  process  is  contained  in  Procedures  for  Instructional 
Systems  Development  (NAVEDTRA  110).  Specifically,  NAVEDTRA  1 1 0  lists  four 
genera)  actions: 
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CRITICALITY  INDEX 


HIGH  MODERATE  LOW 


HIGH 

Optimal  Training 

Moderate 

Overtraining 

Significant 

Overtraining 

MODERATE 

14* 

Moderate 

Optimal 

Moderate 

Undertraining 

Training 

Overtraining 

LOW 

Significant 

Moderate 

Optimal 

Undertraining 

Undertraining 

Training 

*Number  14  refers  to  Job  Task  Statement  Number  14  used  in  sample 
computation  in  figure  A-7. 


Figure  A-6.  Experimental  Training  Assessment 


Computed  training  proficiency  and  utilization  percentages  for  survey  Job  Task 
Statement  Number  14  (Tie  nautical  knots)  (see  figures  A-3  and  A-4): 

Training  Proficiency  Percentage:  64.3 
Utilization  Percentage:  77.8 

Performance  Adequacy  Index  from  table  A-4:  Moderate 
Criticality  Index  from  table  A- 5:  High 

Matrix  Cell  Assessment:  Moderately  Undertrained 

Figure  A-7.  Sample  Derivation  of  Performance  Adequacy 
and  Criticality  Indices 
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•  decide  if  there  is  a  training  problem  based  on  clear-cut 
evidence  of  a  problem 

•  isolate  the  problem  and  determine  its  causes  and  effects 

•  decide  the  best  way  to  solve  the  problem 

•  if  it  is  a  type  A  or  B  change  (i.e.,  a  major  change  (see  CNET 

Instruction  1500.23  series  for  description)),  develop  a 
project  plan  and  request  permission  to  undertake  the  effort. 

The  feedback  data  gathering  procedures  and  data  analysis  methods 
previously  described  are  specifically  related  to  obtaining  evidence 
concerning  the  existence  or  nonexistence  of  training  problems  and  to 
isolate  those  problems  and  their  causes.  This  was  the  nature  and 
limitations  of  the  prototype  tasking.  Additional  techniques  for 
utilizing  the  data  were  not  addressed  directly.  However,  an  experi¬ 
mental  methodology  was  devised  that  is  directed  toward  determining 
solutions  to  the  training  problems  and  the  most  appropriate  point  in 
the  ISD  process  at  which  to  apply  corrective  action.  Attachment  4 
contains  an  extended  worksheet  or  booklet  for  this  purpose.  However, 
implementation  of  actual  changes  in  course  curriculum  requires  necessary 
command  approval  in  accordance  with  pertinent  instructions  and  directives. 
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ATTACHMENT  1 

GUIDELINES  FOR  PREPARING 
FEEDBACK  INSTRUMENT  TASK  STATEMENTS 

This  attachment  contains  guidelines  for  writing  job  task  statements  for 
use  on  feedback  forms.  Portions  of  these  guidelines  are  based  on  TAEG 
Technical  Note  4-79. 
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1 .  use  abbreviations  cautiously  since  they  may  not  be  understood  by  all 
individuals  responding  to  a  task  statement.  It  is  good  practice  to  spell  out 

a  term  when  it  first  appears  and  follow  it  with  the  abbreviation  in  parenthesis. 
In  subsequent  tasks,  the  abbreviation  may  stand  alone. 

2.  Use  short  words  and  phrases  in  ^reference  to  long  words  or  expressions. 

for  example,  "Write  production  and  control  reports"  is  preferred  over  "Accomplish 
necessary  reports  involved  in  the  process  of  maintaining  production  and  control 
procedures . " 

3.  Begin  the  task  statement  with  a  present  tense  action  verb  with  the  subject 
"I"  understood;  for  example,  use  "operate,"  "write,"  "clean,"  instead  of 
"operates,"  "writes,"  "cleans." 

4.  Begin  each  task  statement  with  an  action  verb  which  specifies  behavior 
that  a  supervisor  can  observe.  Do  not  use  verbs  which  reflect  unobservable 
behaviors  (e.g.,  “plan,"  "devise"). 

Example :  "Plan  troubleshooting  electrical  malfunction  on  aircraft  armament 
system,"  should  be  restated  ir  a  form  such  as: 

"Troubleshoot  electrical  malfunctions  of  aircraft  armament  systems...." 

5.  Related  to  the  above  guidelines  is  the  requirement  to  use  action  verbs 
which  reflect  behaviors  which  the  supervisor  can  observe  in  the  on-the-job 
performance  of  a  graduate.  Avoid  using  verbs  such  as  "describe,"  "explain," 
which  would  require  the  supervisor  to  question  a  graduate  before  he  could  rate 
training  adequacy. 

Example:  "Describe  safety  precautions  involved  when...."  is  not  considered 
appropriate  for  evaluation  questionnaire  use.  Where  possible, 
such  statements  should  be  converted  to  forms  such  as: 

"Use  (ooserve,  practice,  follow)  proper  safety  precautions  involved 
when. ..." 

6.  Make  each  task  statement  specific  and  capable  of  standing  alone.  Do  not 
use  an  action  subheading  followed  by  a  series  of  objects. 

Example :  "Operate  the  following  equipment:  (1)  Automatic  capsule  filler, 

(2)  distilling  apparatus,  '3)  force  filters"  should  be  restated 
separately  in  a  form  such  as: 

"(1)  operate  automatic  capsule  filler,  (2)  operate  distilling 
apparatus,  and  (3)  operate  force  filters." 

7.  Use  simple  statements  without  qualifiers  unless  the  qualifier  is  essential 
to  the  meaning  of  the  statement.  For  example,  "operate  power  mower"  is  preferred 
over  "operate  power  mower  to  cut  grass,"  since  the  qualifier  is  not  necessary. 
However,  "schedule  personnel  for  formal  training"  is  preferred  over  "schedule 
personnel . " 

8.  If  a  modifier  is  needed  for  greater  specificity,  be  sure  to  include  all 
other  significant  tasks  with  comparable  modifiers.  For  example,  in  an  automotive 
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mechanic  inventory,  "repair  transmissions"  would  probably  be  specific  enough. 
However,  if  the  statement  were  modified  to  read  "repair  automatic  transmissions," 
"repair  standard  transmissions"  should  also  be  added  to  the  list. 

9.  Avoid  stating  tasks  that  are  obviously  too  specific  or  trivial.  For 
example,  "operate  fork  lift"  is  sufficient.  It  is  not  usually  necessary  to 
list  subordinate  tasks  such  as:  "turn  ignition  key,"  "shift  gears," 

"elevate  fork." 

10.  Avoid  listing  tasks  that  are  too  general.  For  training  evaluation  purposes 
task  statements  such  as,  "repair  carburetors,"  "repair  body  sections,"  are 
preferred  over  more  global  statements,  such  as  "repair  motor  vehicles." 

11.  Avoid  redundant  and  unnecessary  qualifying  phrases  such  as  "when  appropriate," 
"as  required,"  "in  accordance  with  prescribed  directives."  For  example,  "main¬ 
tain  logs"  is  probably  sufficient.  Forms  such  as,  "maintain  necessary  logs  in 
accordance  with  prescribed  Navy-wide  or  local  regulations  and  directives" 

would  not  normally  be  necessary  for  an  evaluator  to  understand  what  graduate 
action  he  is  to  evaluate. 

12.  Present  only  one  job  task  at  a  time.  Multiple  tasks  may  occur  from  stating 
more  than  one  verb  requiring  dissimilar  actions  (e.g.,  "remove  and  repair")  or 
more  than  one  object.  Both  are  to  be  avoided  since  the  supervisors  opinion  of 
training  adequacy  or  appropriateness  may  not  be  the  same  for  all  tasks.  Separate, 
discrete  statements  are  preferred  as  shown  in  the  two  examples  below. 

Example  1 :  "Use  proper  safety  precautions  involved  when  working  on  both 

energized  and  deenergized  circuits  and  in  the  use  of  general 
cleaning  agents"  should  be  divided  into  statements  such  as: 

•  "Use  proper  safety  precautions  while  working  on  energized 
ci rcui ts. " 

•  "Use  proper  safety  precautions  while  working  on  deenergized 
circuits." 

•  "Use  proper  safety  precautions  while  using  general  cleaning 
agents . " 

Example  2:  "Manually  load,  am,  de-arm,  and  download  inert  airborne  bombs" 

should  be  divided  into  statements  such  as: 

•  "Manually  load  inert  airborne  bombs." 

•  "Manually  arm  inert  airborne  bombs." 

•  "Manually  de-arm  inert  airborne  bombs." 

•  "Manually  download  inert  airborne  bombs." 

13.  Group  more  than  one  job  task  only  when  they  are  usually  done  simultaneously 
or  as  a  part  of  one  general  evolution. 

Example:  "Measure  AC  and  DC  voltage,  small  DC  current  and  resistance  with 

a  multimeter  AN  PSM-4  or  an  equivalent." 
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14.  Include  the  equipment  needed  to  perform  a  task  if  there  are  other  types 
of  equipment  with  which  to  do  the  task. 

Example:  "Measure  resistance  of  insulation  with  a  500  volt  meggar." 

15.  Do  not  generate  job  task  statements  that  primarily  relate  to  skills  taught 
in  the  school  to  assist/reinforce  theoretical  understanding  and  which  reflect 
tasks  infrequently  done  on  the  job. 

16.  Avoid  referencing  publications  by  number.  Fleet  supervisors  may  not  "know" 
the  document(s)  or  be  intimately  aware  of  its  contents.  It  is  not  desirable  to 
require  a  supervisor  to  look  up  publications  for  completing  a  feedback  instrument. 

Example:  "Describe  equipment  tag-out  procedures  in  accordance  with 

OPNAVINST  3120.32"  can  be  restated  in  a  form  such  as: 

"Tag-out  equipment." 

17.  Generate  job  task  statements  which  can  be  expressed  clearly  and  concisely 

in  behavioral  terms.  The  following  potential  evaluation  instrument  item  probably 
should  be  discarded  or  a  determination  made  of  the  essential  behaviors  involved 
in  the  performance  of  this  job. 

"Perform  procedural  steps  as  stated  in  NAVEDTRA  43241 B  for  Theory 
Qualification  101,  except  201.218  and  Watchstanders  Qualification 
401  as  applicable." 

18.  Avoid  writing  task  statements  that  are  so  equipment  specific  that  they 
are  likely  to  produce  large  numbers  of  "not  observed"  responses.  Create  more 
general  statements  when  it  is  necessary. 

Example:  "Select  and  execute  the  proper  utility  routines  for  job  execution 

on  the  AN/UYK-S(V)  as  outlined  in  the  V- 1 500  SMIS  Software 
Manual"  should  be  restated  in  a  form  such  as: 

"Select  and  execute  the  proper  utility  routines  for  job  execution 
on  the  assigned  computer." 
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ATTACHMENT  2 

INTERVIEW  KIT 

This  attachment  contains  an  Interview  Kit  (i.e.,  samples  of  forms  recommended 
for  use  in  a  school  based  training  feedback  collection  program  along  with  instruc¬ 
tions  for  their  use).  This  attachment  is  comprised  of  four  annexes.  Annex  A 
contains  sample  pages  of  an  interview  form  which  provides  an  example  of  the 
recoimended  format.  A  set  of  instructions  for  the  use  of  the  suggested  interview 
form  is  contained  in  annex  B.  Annex  C  is  a  sample  interviewee  Background  Data 
Form.  Annex  D  lists  possible  nonperformance  and  performance  deficiency  Reason 
Codes  for  use  during  interviews. 
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ANNEX  A 

SAMPLE  INTERVIEW  FORM 

This  annex  contains  four  pages  of  a  sample  interview  form.  The  first  page 
is  a  title  page.  The  second  page  is  the  first  data  collection  page.  It  includes 
instructions  to  the  interviewee.  The  third  page  is  a  typical  data  collection 
page,  and  the  fourth  is  the  final  "general  comments"  page  of  the  interview  form. 
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ANNEX  B 

SAMPLE  INTERVIEW  INSTRUCTIONS 


This  annex  contains  suggested 
includes  preinterview  guidance  for 
the  interviewee. 


instructions  for  conducting  interviews.  This 
the  interviewers  and  directions  to  be  read  to 
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INTERVIEWER  GUIDANCE 
INTRODUCTION  AND  PURPOSE  OF  PROJECT 

The  purpose  of  this  effort  is  to  obtain  firsthand  information  from  personnel 
newly  arrived  from  fleet  units  who  had  occasion  to  observe  and  evaluate  recent 
graduates  of  their  respective  "A"  schools. 

GENERAL  INFORMATION  ABOUT  THE  INTERVIEW 

1.  Insure  only  one  interview  is  scheduled  at  a  time.  Be  sure  to  have  a 
private  area  available  for  your  use.  The  Interviewing  Package  should  contain 
the  following: 

•  Directions  to  Be  Read  to  the  Interviewee 

•  Interview  Form 

•  Two  copies  of  Reason  Codes  (one  for  you  and  one  for  the  interviewee). 

2.  When  the  person  reports  to  you,  read/paraphrase  the  following  infor¬ 
mation  to  him/her.  (This  will  explain  the  purpose  of  the  project  and  what  is 
expected  of  the  student.) 

"The  purpose  of  this  project  is  to  obtain  information  on  how 
effectively  'A'  schools  are  meeting  the  needs  of  the  fleet.  The  way 
we  are  doing  this  is  to  interview  persons  like  yourself  who  have 
observed  or  supervised  recent  graduates  from  the  ‘A1  school  after 
they  reported  to  the  fleet.  8ecause  of  your  unique  experience  your 
input  is  the  best  source  of  information  available  for  this  purpose. 

Any  data  reported  will  be  grouped  in  tables.  Individual  names 
will  not  be  used.  Your  thoughtful  and  accurate  answers  are 
needed. " 

3.  Go  over  the  instructions  for  Part  A  of  the  interview  form  with 
the  individual  to  be  sure  he/she  understands  what  is  required. 

4.  Instruct  the  individual  to  complete  Part  A. 

5.  When  you  receive  the  interview  form  from  the  student,  review 

Part  A  quickly  to  be  sure  a  category  has  been  selected  for  each  task.  You  will 

fill  in  Part  B  as  you  conduct  the  interview.  Go  through  all  of  the  "Don’t  Do" 

tasks  first.  Have  the  interviewee  select  a  reason  code  from  the  "Don’t  Do 
category.  After  the  appropriate  code  is  selected,  ask  for  and  record  the 
specific  information  that  prompted  the  selection.  If  none  of  the  codes  are 
applicable,  write  the  reason  in  the  space  provided.  If  more  space  is  needed, 
write  on  the  back  of  the  page  with  the  identifying  task  number.  Then  go  back 
to  the  tasks  identified  as  "Do  With  Difficulty."  Again,  ask  the  respondent 
to  select  an  appropriate  reason  code.  Obtain  additional  information  as  to 
why  It  is  difficult.  These  data  are  the  most  important  of  the  survey.  Be  as 
thorough  as  possible. 

For  those  tasks  checked  "Do  With  Ease,"  ask  the  interviewee  if  there  are 
any  tasks  which  he  would  like  to  discuss  or  explain.  (Consider  here  correct 
"A"  school  curriculum  emphasis,  possible  instances  of  overtraining,  and  areas 
for  elimination  from  the  curriculum.) 
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6.  Complete  the  interview  using  the  last  page  of  the  interview  form. 

This  page  requests  the  interviewee  to  consider  tasks  which  are  not  taught  in  "A" 
school  but  should  be  taught  there.  It  also  asks  for  tasks  the  student  has  had 
to  learn  on  the  job,  specifically  tasks  that  cannot  be  performed  because  they 
were  not  learned  in  "A"  school,  and  for  which  training  is  not  available  on  the 
job. 


7.  When  you  conduct  the  interview,  follow  the  "Directions  to  Be  Read  to 
the  Interviewee."  This  will  help  to  insure  standardization  of  the  procedures 
which  is  important  in  interpreting  the  data. 

Record  the  time  it  took  to  conduct  the  interview. 

Thank  the  student  for  his/her  time  and  effort. 
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DIRECTIONS  TO  BE  READ  TO  THE  INTERVIEWEE 

DURING  THIS  INTERVIEW,  I  WILL  GO  OVER  YOUR  RESPONSES  TO  THE  TASK  STATE¬ 
MENTS  WITH  YOU,  AND  ASK  YOU  TO  PROVIDE  MORE  INFORMATION  ABOUT  THE  ABILITY  OF 
"A"  SCHOOL  GRADUATES  TO  PERFORM  JOB  TASKS. 

FOR  THOSE  TASK  STATEMENTS  YOU  MARKED  "DON'T  DO,"  I  WILL  READ  EACH  STATE¬ 
MENT  THUS  MARKED.  I  WOULD  LIKE  YOU  TO  EXAMINE  THE  "DON'T  DO"  SECTION  OF  THE 
REASON  CODE  SHEET.  AFTER  I  READ  EACH  ITEM,  PICK  THE  REASON  (OR  REASONS)  THAT 
BEST  DESCRIBE  WHY  YOU  ANSWERED  "DON'T  DO"  TO  THAT  ITEM.  IF  NONE  OF  THE  CODES 
ARE  APPLICABLE,  TELL  ME  THE  REASON  WHY  YOU  MARKED  THE  ITEM  THAT  WAY.  (Ask  for 
amplification  of  the  coded  answer;  be  specific.)  (Work  through  all  the 
"Don't  Do's."  When  finished,  read  the  material  below  to  the  interviewee.) 

NOW  LET'S  PROCEED  TO  THE  "DO  WITH  DIFFICULTY"  ITEMS.  AFTER  I  READ  EACH 
ITEM,  DO  THE  SAME  THING  YOU  JUST  DID,  BUT  PICK  YOUR  CODES  FROM  THE  "DO  WITH 
DIFFICULTY"  SECTION.  (Ask  for  amplification  of  the  coded  answer;  be  specific.) 
(After  finishing  the  "Do  With  Difficulty"  list,  discuss  the  "Do  With  Ease" 
items. ) 

DO  YOU  HAVE  ANY  GENERAL  COMMENTS  ABOUT  THE  "DO  WITH  EASE"  ITEMS?  FOR 
EXAMPLE,  HOW  WELL  IS  THE  SCHOOL  TEACHING  THEM?  ARE  THERE  ANY  THAT  MAY  REPRE¬ 
SENT  INSTANCES  OF  OVERTRAINING  OR  WOULD  YOU  RECOMMEND  DROPPING  ANY  OF  THEM 
FROM  THE  SCHOOL  CURRICULUM?  (Record  his/her  answers.) 

(Complete  the  interview  by  going  to  the  last  page  of  the  Interview  form. 
Ask  the  interviewee  for  any  further  suggestions  or  recommendations.) 

(Thank  the  interviewee  for  his  cooperation  and  effort  in  completing  this 
interview. ) 
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ANNEX  C 

SAMPLE  BACKGROUND  DATA  FORM 

This  annex  contains  a  sample  Background  Data  Form  for  collecting  infor¬ 
mation  about  individuals  attending  advanced  schools  and  for  selecting 
interviewees. 
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BACKGROUND  DATA  FORM 


1.  NAME _ RATE  _ 

2.  TIME  IN  SERVICE  (ACTIVE  DUTY)  IN  YEARS  _ 

3.  COURSE  ATTENDING  _ 

4.  WHAT  WAS  YOUR  PREVIOUS  DUTY  STATION?  SEA  DUTY  _  SHORE  DUTY  _ 

5.  IF  YOU  CAME  FROM  SHORE  DUTY,  WHERE  WERE  YOU  STATIONED?  _ 

6.  IF  YOU  CAME  FROM  SEA  DUTY,  WHAT  SHIP  CLASS  (TYPE)  WERE  YOU  ASSIGNED  TO? 


7.  WHAT  WAS  YOUR  PRIMARY  BILLET  TITLE?  _ 

8.  IN  THE  LAST  12  MONTHS,  HAVE  YOU  HAD  AN  OPPORTUNITY  TO  OBSERVE  AND  EVALUATE 
RECENT  GRADUATES  OF  CLASS  "A"  SCHOOL  IN  AN  OPERATIONAL  ASSIGNMENT? 

YES  _  NO  _ 

IF  YOU  ANSWERED  "YES"  TO  THE  PREVIOUS  QUESTION,  PLEASE  ANSWER  THE  FOLLOWING 
QUESTIONS.  IF  YOU  DID  NOT  ANSWER  "YES,"  STOP  AND  DISCUSS  YOUR  ANSWER  WITH 
THE  INSTRUCTOR. 

9.  HOW  MANY  PEOPLE  DID  YOU  OBSERVE  AND  EVALUATE?  _ FOR 

HOW  LONG?  _ _ 

(Number  of  Months) 

10.  HOW  SATISFIED  WERE  YOU  WITH  THEIR  ABILITY  TO  MEET  MINIMUM  JOB  REQUIREMENTS 
AFTER  A  TYPICAL  BREAK-IN  PERIOD? 

HIGHLY  SATISFIED 

MODERATELY  SATISFIED 

SATISFIED 

MODERATELY  UNSATISFIED 
HIGHLY  UNSATISFIED 

11.  IF  YOU  ANSWERED  UNSATISFIED  TO  ANY  DEGREE,  LIST  THE  GENERAL  AREA(S)  THAT 
REQUIRED  ADDITIONAL  TRAINING  OVER  AND  BEYOND  THE  TYPICAL  BREAK-IN  AND 
ORIENTATION. 
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ANNEX  D 

SAMPLES  OF  REASON  CODES 

This  annex  contains  40  possible  nonperformance  and/or  performance 
deficiency  Reason  Codes  for  "Don't  Do"  and  "Do  With  Difficulty"  interviewee 
response  choices.  Within  the  "Do  With  Difficulty"  category  Reason  Codes 
are  listed  for  three  general  categories  of  reasons  for  performance  deficiency 
operational  factors,  personal  adjustment  problems,  and  technical  training 
inadequacy. 
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REASON  CODES 

"DON'T  DO"  CATEGORY 

A.  TASK  NOT  DONE  ON  MY  SHIP/STATION/SYSTEM 

B.  TASK  NOT  EXPECTED  OF  SOMEONE  IN  RECENT  GRADUATE'S  RATE  OR  LEVEL  OF 
EXPERIENCE 

C.  TASK  EXPECTED  OF  RECENT  GRADUATES  BUT  AVERAGE  GRADUATE  UNABLE  TO  PERFORM 

D.  NOT  SUFFICIENT  TIME  FOR  GRADUATE  TO  HAVE  BEEN  ASSIGNED  TO  THIS  TASK. 

"DO  WITH  DIFFICULTY"  CATEGORY 

OPERATIONAL 

E.  TIME  CONSTRAINTS  DUE  TO  OPERATIONAL  COMMITMENTS 

F.  FATIGUE/STRESS  DUE  TO  OPERATIONAL  COMMITMENTS 

G.  EQUIPMENT  EXTREMELY  COMPLEX 

H.  TECHNICAL  MANUALS  TOO  COMPLEX 

I.  INEFFECTIVE  LEADERSHIP/SUPERVISiON 

J.  DIFFICULT  ACCESS  TO  EQUIPMENT 

K.  TASK  HIGHLY  UNPLEASANT  (i.e.,  uncomfortable  environmental  situation 

such  as  heat,  soot,  shock  hazard,  exposure 
to  elements). 

PERSONAL 

L.  LOW  MOTIVATION  FOR  OPERATIONAL  ASSIGNMENT 

M.  LOW  MOTIVATION  FOR  WORK  CENTER  ASSIGNMENT 

N.  LOW  MOTIVATION  FOR  TECHNICAL  SPECIALTY 

O.  LOW  MOTIVATION  FOR  NAVAL  SERVICE 

P.  INTERFERENCE  FROM  PERSONAL  PROBLEMS  (Legal/Psychological/Health/ 

Substance  Abuse). 

TECHNICAL 

0.  DID  NOT  KNOW  WHAT  TO  LOOK  FOR  DURING  INSPECTION  OF  SYSTEM  (OR  COMPONENT) 

R.  DID  NOT  UNDERSTAND  STEPS  OF  PROCEDURE  ON  MRC_ 

S.  COULD  NOT  "READ"  PRINT  IN  EQUIPMENT  TECHNICAL  MANUAL  (INSTRUCTION  BOOK) 
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REASON  CODES  (continued) 

T.  DID  NOT  KNOW  WHICH  TOOL  OR  TEST  EQUIPMENT  TO  USE  (SELECTED  WRONG  TOOL  OR 
TEST  EQUIPMENT  FOR  JOB) 

U.  DID  NOT  KNOW  WHICH  MATERIAL  TO  USE  (SELECTED  WRONG  MATERIAL  FOR  SERVICE 
USE) 

V.  COULD  NOT  IDENTIFY  WHICH  PART  CAUSED  MALFUNCTION  (IDENTIFIED  WRONG  PART) 

W.  NOT  WELL  VERSED  IN  TROUBLESHOOTING  PRINCIPLES 

X.  COULD  NOT  USE  TOOLS  OR  TEST  EQUIPMENT  CORRECTLY 

Y.  COULD  NOT  DISASSEMBLE  (OR  REASSEMBLE)  COMPONENT 

Z.  COULD  NOT  PERFORM  PROCEDURAL  STEPS  ON  MRC  CORRECTLY 
AA.  COULD  NOT  MAKE  REQUIRED  MEASUREMENTS  (OR  ADJUSTMENTS) 

BB.  COULD  NOT  PERFORM  REQUIRED  TESTS 

CC.  COULD  NOT  FABRICATE  REPLACEMENT  PART(S) 

DD.  COULD  NOT  IDENTIFY  ACTUAL  COMPONENTS  OF  SYSTEM  (OR  PARTS  OF  COMPONENTS) 

EE.  COULD  NOT  EXPLAIN  FUNCTION  (OR  PURPOSE)  OF  COMPONENT(S)  IN  SYSTEM 
FF.  COULD  NOT  '‘READ"  SYSTEM  SCHEMATIC  AND  WIRING  DIAGRAM 
GG.  COULD  NOT  "READ"  SYSTEM  PIPING  DIAGRAM 

HH.  COULD  NOT  EXPLAIN  OPERATING  PRINCIPLE (S)  OF  SYSTEM  (OR  COMPONENT) 

II.  COULD  NOT  TELL  WHEN  SYSTEM  (OR  COMPONENT)  HAD  A  MALFUNCTION 
JJ.  COULD  NOT  PHYSICALLY  "TRACE"  SYSTEM  FROM  ONE  POINT  TO  ANOTHER 
KK.  COULD  NOT  LINE  UP  SYSTEM  (OR  COMPONENT)  IN  ACCORDANCE  WITH  INSTRUCTIONS 
LL.  COULD  NOT  OPERATE  TEST  EQUIPMENT  PROPERLY 

MM.  COULD  NOT  STOP/SECURE  SYSTEM  (OR  COMPONENT)  IN  ACCORDANCE  WITH  INSTRUCTIONS 
NN.  CAN  DO  THE  TASK  BUT  NOT  AS  QUICKLY  AS  NECESSARY/DESIRED. 
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ATTACHMENT  3 

GUIDANCE  FOR  CONDUCTING  TRAINING  APPRAISAL  INTERVIEWS 

This  attachment  contains  the  complete  text  of  a  U.S.  Air  Force  publica¬ 
tion  entitled  The  Evaluation  Interview.  The  document  (undated)  was  prepared 
by  the  USAF  3415th  Technical  School  (now  the  3400th  Technical  Training  Wing), 
Lowry  Air  Force  Base,  Colorado.  It  contains  a  concise  review  of  interview 
techniques  applied  directly  to  acquiring  training  appraisal  information  in 
a  military  setting.  Because  of  its  precise  focus  and  relevance  to  the 
previously  recommended  interview  procedures,  the  entire  content  is  included. 
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THE  EVALUATION  INTERVIEW 

There  are  many  ways  to  evaluate  training,  however  two  of  the  most 
common  are  interviews  and  questionnaires.  It  is  not  unusual  to  hear,  "Ques¬ 
tionnaires  are  no  good,  that  interview  of  graduates  is  best."  Unfortunately, 
the  method  of  conducting  the  interview  may  be  such  that  the  interview  results 
are  no  better  than  those  obtained  from  the  much  maligned  questionnaire. 

The  most  common  interview  is  the  question-answer  type.  An  observation- 
interview  may  be  conducted  to  actually  observe  the  graduate  as  he  performs 
the  tasks  and  jobs  of  his  specialty.  Neither  of  these  types  of  interview  is 
easy  to  conduct.  Properly  conducted,  each  will  produce  reliable  and  valid 
information. 

In  any  form,  the  interview  is  a  means  of  collecting  information.  To  the 
extent  you  are  obtaining  information,  each  time  you  ask  a  question  you  are 
interviewing.  A  conference  is  a  group  interview.  We  spend  hours  preparing 
for  a  conference  and  minutes  in  preparing  for  a  person  to  person  interview. 

In  both  cases  the  objective  is  to  collect  information  and  in  both  cases  the 
amount  of  information  collected  is  directly  proportional  to  the  preparation 
and  effort  expended. 

Because  of  the  versatility  and  flexibility  of  the  interview,  it  has  a 
particular  value  in  an  evaluation  program.  The  interview  permits  exploring 
undocumented  areas  of  information  specific  to  the  training.  It  allows  extend¬ 
ing  the  investigation  while  it  is  still  in  progress  and  most  important,  it 
provides  for  exploration  of  related  information  at  the  most  propitious  time. 

Evaluation  usually  requires  inquiries  into  the  personal  opinions,  desires, 
satisfactions,  and  fears  of  the  graduate.  The  interview  is  invaluable  in 
these  situations  because  it  allows  the  evaluator  to  probe  the  graduate's 
innermost  thoughts. 

With  all  of  its  touted  advantages,  the  interview  has  some  disadvantages; 
principally  time,  cost,  and  the  difficulty  of  obtaining  the  qualified  inter¬ 
viewer.  Some  of  these  disadvantages  can  be  minimized  by  proper  planning. 

Both  time  and  cost  factors  can  be  made  more  acceptable  by  selecting  a  locale 
having  a  large  graduate  population  available  for  interviewing.  The  securing 
of  a  qualified  interviewer  is  not  so  easily  solved.  It  is  the  purpose  of  the 
remainder  of  this  article  to  suggest  to  the  evaluator-interviewer  some  of  the 
techniques  which  will  help  him  to  obtain  the  information  he  seeks.  Procedures 
are  equally  appropriate  when  interviewing  supervisors,  or  for  any  type  of 
interviewing. 

With  this  introduction  to  the  techniques  of  interviewing,  it  is  appro¬ 
priate  to  discuss  the  preparations  for  the  interview. 

The  evaluator  planning  to  visit  a  base  for  the  purpose  of  interviewing 
graduates  should  request  suitable  facilities  in  which  to  conduct  the  inter¬ 
views.  He  is,  of  course,  completely  dependent  upon  the  judgment  of  his  base 
contact  as  to  what  is  suitable,  and  upon  the  facilities  available  at  the 
base. 
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The  first  act  in  an  interview  must  be  an  introduction  of  who  you  are  and 
why  you  are  conducting  the  interview.  Your  name  in  itself  may  be  meaningless 
to  the  graduate,  but  it  is  important  that  your  introduction  be  given  clearly, 
don't  mumble.  It  is  equally  important  that  you  learn  the  graduate's  name  and 
that  you  pronounce  it  correctly.  If  you're  not  sure  of  the  pronunciation  ask 
what  is  correct.  It  is  embarrassing  to  try  to  talk  to  people  without  knowing 
their  name.  Names  are  important  to  people  and  they  are  pleased  and  flattered 
when  a  comparative  stranger  has  enough  interest  to  learn  their  name  and 
pronounce  it  correctly. 

When  the  introductions  are  completed  you  should  explain  in  some  detail 
what  you  expect  to  learn  from  the  graduate  and  what  you  intend  to  do  with 
that  information.  Once  again,  be  alert  and  observant.  From  this  point  to 
the  conclusion  of  the  interview,  every  action  and  every  word  uttered  by  the 
graduate  has  meaning;  a  meaning  you  must  interpret  correctly  if  your  inter¬ 
view  is  to  be  successful. 

Your  preparations  are  now  complete  and  you  are  ready  to  delve  into  the 
technical  aspects  of  the  information  you  must  collect  in  the  interview.  To 
do  this,  you  need  to  know  more  about  the  techniques  of  interviewing.  Basically 
you  may  select  any  of  three  interviewing  techniques,  but  it  is  almost  certain 
you  will  combine  them  during  the  course  of  an  interview.  One  more  caution, 
make  voluminous  notes.  A  good  interview  develops  many  ideas,  many  answers, 
far  more  than  you  can  remember  when  you're  back  at  the  office.  Note  key 
words  and  ideas  during  the  interview,  expand  these  into  a  complete  report  at 
the  conclusion  of  the  interview. 

If  there  is  a  need  for  precise  comparison  of  the  answers  given  by  all  of 
the  graduates  to  all  of  the  questions,  an  in-depth  schedule  of  questions  must 
be  prepared.  The  interviewer  will  read  these,  word  for  word,  and  in  the 
exact  sequence,  to  the  graduate  and  record  his  answers.  This  procedure 
assures  that  identical  questions  are  asked  in  exactly  the  same  sequence  of 
each  graduate  and  because  of  this,  each  answer  by  each  graduate  can  be  com¬ 
pared  with  all  the  answers  from  all  of  the  graduates. 

When  less  precise  comparison  is  sought,  the  schedule  of  questions  is 
prepared  for  guidance.  The  interviewer  may,  at  his  discretion,  reword  or 
rephrase  the  questions,  change  the  sequence,  add  more  questions,  or  even 
delete  questions,  in  an  effort  to  match  the  climate  of  the  interview  to  the 
mood  of  the  graduate.  In  this  type  interview  the  schedule  of  questions 
assures  general  coverage  of  the  material. 

Finally,  there  is  the  completely  open  interview.  The  interviewer  asks  a 
provocative  question  and  thereafter  does  little  more  than  encourage  the 
graduate  to  talk.  This  type  is  difficult  to  use,  but  produces  extremely 
valuable  information  available  from  no  other  source. 

At  some  time  during  the  progress  of  an  interview,  the  skilled  interviewer 
will  make  use  of  each  of  these  techniques.  The  skill  with  which  he  uses  them 
will,  to  a  large  extent,  determine  the  quantity  and  value  of  the  information 
he  obtains  from  the  graduate.  In  all  interviewing  there  is  a  minimum  of 
information  which  is  acceptable  and  a  maximum  which  is  possible.  No  inter- 
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view  should  fail  to  produce  the  minimum  and  no  interview  should  have  less 
than  the  maximum  for  its  goal. 

It  is  highly  important  that  the  schedule  of  questions  be  carefully 
prepared.  When  the  evaluator  is  not  technically  qualified  in  the  work  area 
he  would  be  wise  to  enlist  the  assistance  of  a  technician.  If  he  prepares 
the  questions  himself,  they  should  be  checked  by  a  technician. 

During  the  early  part  of  tne  interview  the  evaluator  should  stay  within 
the  schedule  of  questions,  starting  by  reading  the  questions  verbatim  and 
keeping  the  same  sequence.  The  somewhat  formal  atmosphere  and  the  accuracy 
of  the  questions  tends  to  increase  the  graduate's  confidence  in  the  evaluator 
as  knowing  the  field  of  work.  As  the  interview  progresses,  the  evaluator 
must  be  alert  to  indications  that  the  graduate  wants  to  volunteer  information 
beyond  the  scope  of  the  questions. 

When  it  becomes  evident  that  the  graduate  is  anxious  to  tell  his  story, 
the  interview  should  shift  to  the  second  technique,  asking  questions  as  they 
are  cued  by  the  answers  and  comments  of  the  graduate.  It  is  desirable  at 
this  stage  to  stay  within  the  parameters  of  the  schedule  of  questions. 

If  the  interview  has  been  conducted  skillfully  to  this  moment,  and  if 
the  time  is  available,  the  third  technique  should  be  employed.  The  third 
technique  is  useful  for  determining  concepts,  attitudes,  and  personal  attri¬ 
butes.  It  can  be  used  to  explore  the  intangibles  so  important  to  a  full 
understanding  of  the  graduate's  problems,  but  so  completely  separate  from  the 
technically  oriented  portion  of  the  interview.  The  graduate  is  ready  for 
this  technique  when  he  begins  to  insert  personal  comments  among  his  technical 
answers. 

The  open  interview  is  often  referred  to  as  spontaneous,  and  it  should 
give  this  impression.  However,  it  is  not  spontaneous,  it  must  be  planned. 

The  interviewer  must  have  in  his  repertoire  carefully  developed  questions 
designed  to  elicit  the  personal  type  answer.  Planning  here  is  vital,  but  it 
must  never  show  except  in  the  results.  This  type  interview  tests  the  skill 
of  the  interviewer  to  the  utmost. 

Timing  is  the  secret  weapon  of  the  interviewer.  Every  speaker,  every 
actor  knows  the  value  of  timing  to  his  performance.  Jack  Benny  and  Bob  Hope 
are  experts  in  the  use  of  timing  and  both  exploit  the  "pause"  to  its  full 
limit.  The  pause  is  equally  valuable  in  interviewing,  following  a  pithy 
comment  or  the  posing  of  a  penetrating  question.  Don’t  be  afraid  to  wait  for 
the  graduate  to  comment  or  answer  your  question.  You  planned  this  interview, 
he  didn’t  and  he  needs  time  to  think. 

Generally  speaking,  the  answers  the  graduate  gives  to  your  questions 
will  contain  clues  to  the  area  of  questioning  that  should  follow.  Avoid 
introducing  questions  that  have  not  been  cued  by  some  previous  reference. 

This  is  a  part  of  the  proper  timing  for  asking  a  question.  Be  alert  to 
openings  presented  by  the  graduate  as  he  rambles  from  subject  to  subject,  but 
still  within  the  purview  of  the  interview.  In  some  cases,  the  point  must  be 
forcibly  exploited,  but  usually  it  is  better  to  lead  the  answers  toward  the 
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area  in  which  you  are  seeking  information.  Timing  has  different  implications 
for  the  observation  type  interview,  but  is  equally  important  to  the  success  of 
the  interview.  The  interviewer  must  never  pose  a  question  while  tne  graduate 
is  actually  performing  a  task.  First,  to  do  so  may  cause  an  accident  by 
dividing  the  graduate's  attention.  And,  second,  dividing  the  graduate's 
attention  will  result  in  a  poor  performance.  Timing  is  important,  use  it  to 
improve  the  results  of  your  regular  interview  and  during  all  observation- 
i nterviews. 

An  observation-interview  requires  the  evaluator  to  be  present  during  the 
performance  of  the  specialty  tasks  by  the  graduate.  It's  a  lot  easier  to 
"talk  a  good  game  than  it  is  to  play  one. 11  You  can  read,  or  even  formulate 
acceptable  questions,  without  being  a  job  expert,  but  for  the  observation- 
interview  you  must  be  able  to  identify  what  you  see  and  know  why  it  is  being 
done.  An  observer  who  is  familiar  with  the  task  or  job  being  done  is  able  to 
evaluate  both  the  knowledge  being  applied  to  the  performance  of  the  task  or 
job,  and  the  skill  with  which  the  manual  movements  are  made.  The  technically 
qualified  evaluator  is  also  able  to  estimate  the  quality  of  what  has  been 
produced. 

An  important  part  of  the  ability  to  observe  accurately  is  being  able  to 
identify  the  various  work  activities  as  they  occur  and  know  what  knowledges 
and  skills  are  being  exhibited.  Close  attention  to  details  is  essential  to 
accurate  observation.  If  the  graduate  shows  small  hesitations,  makes  false 
movements,  or  looks  to  see  who  is  watching,  he  is  plagued  by  inadequacy  and 
indecision. 

The  observation  must  include  such  items  as  job  organization,  selection 
and  placement  of  tools,  parts,  materials,  and  equipment,  and  the  choosing  of 
the  correct  procedures.  All  of  these  are  indicators  of  the  graduate's  overall 
knowledge  of  the  job  arid  his  proficiency  during  his  performance. 

When  the  graduate  is  working,  his  movements  should  exhibit  an  observable 
rhythm  and  purpose.  If  his  movements  are  jerky,  lack  of  coordination,  or  he 
is  often  out  of  position,  either  ne  is  improperly  taught,  or  he  has  not 
learned  how  to  perform  the  tasks.  Finally,  the  end  product  of  his  efforts 
must  be  usable,  must  meet  the  established  standards  of  quality.  For  the 
evaluator,  there  is  no  better  advice  than,  "Know  the  job  before  critiquing  the 
performance. " 

Regardless  of  the  interview  technique,  question  and  answer  or  observation 
questions  must  be  developed,  properly  constructed,  and  wisely  used. 

To  some  extent  all  of  the  graduate's  answers  will  be  influenced  by  the 
manner  in  which  the  question  is  asked.  The  phrasing,  the  choice  of  words,  and 
even  the  voice  intonations  may  cause  significant  effects  in  the  graduate's 
answers:  Fundamentally,  there  arc-  two  types  of  questions  used  in  interviewing 

A  type  of  question  sometimes  called  "closed"  or  "limiting",  is  largely 
directive  and  is  answerable  in  a  few  words.  There  are  three  subtypes  of  this 
question.  One  asks  for  identification  of  who,  where,  when,  how,  or  which. 

The  second  asks  for  a  selection  from  among  several  answers  offered.  The  third 
asks  only  for  a  yes  or  no  answer. 
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The  so-called  "open"  or  "nonlimiting"  question  does  not  direct  an  answer, 
rather  it  invites  longer  and  more  descriptive  answers. 

The  evaluator  must  decide  the  ratio  of  "limiting"  to  "non! imi ti ng" 
questions  he  will  use  in  the  interview.  Too  many  "limiting"  questions  may 
cause  the  graduate  to  feel  he  is  being  restricted,  that  he  is  being  prevented 
from  giving  all  of  the  information  he  possesses.  Because  he  feels  this  way 
he  may  sulk  and  withhold  valuable  information.  On  the  other  hand,  too  many 
"nonlimiting"  questions  may  cause  the  interview  to  bog  down  for  lack  of 
direction.  This  is  very  likely  to  happen  if  the  graduate  has  difficulty 
expressing  his  ideas.  Such  people  are  often  inarticulate  and  incoherent. 

During  the  questioning  they  become  greatly  disturbed  and  may  refuse  to  co¬ 
operate.  These  people  work  better  with  the  "limiting"  question  because  it 
provides  a  clue  to  the  desired  answer  to  the  question,  a  question  they  may 
not  have  fully  understood. 

Either  type  question  may  have  a  relationship  with  previous  and  future 
questions,  neither  will  disrupt  the  continuity  of  the  questioning.  The 
evaluator  may  plan  for  a  gradual  build-up  of  blocks  of  information  which  will 
finally  portray  a  complete  job  story.  If  this  is  the  plan,  there  must  be 
several  varieties  of  questions  asked  to  obtain  the  information. 

A  follow-on  style  question  can  be  formulated  asking  for  additional 
information  about  a  subject  previously  discussed.  Another  variety  of  questions 
which  is  quite  useful  is  the  repeating  of  the  graduate's  words.  It  shows 
retention  of  what  he  has  said  and  proves  you  were  listening.  It  can  also  be 
used  to  introduce  a  new  direction  of  questioning. 

When  there  is  a  need  for  clarification  of  vague  or  ambiguous  references 
made  earlier  in  the  interview,  or  when  there  has  been  a  general  reference  and 
there  is  now  a  need  for  explicit  information,  a  specific  variety  of  questions 
must  be  developed. 

It  is  sometimes  necessary  to  summarize  the  information  given  so  that  it 
can  be  confirmed  and  questions  can  be  designed  to  accomplish  this  purpose. 

When  inconsistencies  develop  between  answers  given,  or  between  answers  and 
known  information,  a  direct  question  pointing  out  the  inconsistencies  and 
demanding  correction  is  necessary.  It  is  sometimes  desirable  to  ask  certain 
questions  the  second  time  to  determine  if  the  graduate  gives  the  same  answer. 

A  bit  of  trickery  may  be  used  to  obtain  further  clarification.  The 
evaluator  may  deliberately  misstate  a  fact.  Instantly,  the  graduate  will 
correct  the  statement  and  in  the  process,  clarify  and  expand  his  explanation. 

A  deliberate  misquote  of  a  previous  statement  made  by  the  graduate  is  an  even 
greater  spur.  You  have  misquoted  his  statement  and  he  feels  responsible  for 
correcting  the  misinterpretation.  He  is  immediately  called  upon  to  explain 
in  great,  and  enlightening,  detail  that  you  are  mistaken  and  that  in  truth, 
this  is  what  he  said.  All  the  evaluator  need  do  now  is  listen. 

Trickery  may  be  used  to  elicit  clarification,  but  the  trick  of  using 
"leading"  questions  is  not  always  legitimate.  The  "leading"  question,  used 
either  by  accident  or  by  design,  can  dictate  an  answer.  Most  authorities 
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agree  that  an  interviewer  should  avoid  implying  an  answer  to  his  own  questions 
however,  there  are  times  when  the  evaluator  may  want  to  indicate  what  type 
answer  fie  expects. 

'»  examination  of  what  constitutes  a  "leading"  question  indicates  there 
are  two  elements  which  can  be  used  tc  influence  the  answer  to  a  question. 

One  of  these  is  to  indicate  .yj  choice  of  words,  phrasing,  or  voice  intonation 
the  type  answer  expected.  Tap  second  is  to  assume  that  there  are  certain 
behaviors,  ideologies,  and  values  that  are  common  to  all  graduates  partici¬ 
pating  in  tne  interview  cycle. 

A  question  which  tends  to  indicate  the  expected  answer  requires  the 
evaluator  to  have  considerable  knowledge  about  the  graduate,  or  about  the 
subject  matter.  If  the  evaluator  does  not  have  this  information,  but  uses 
some  method  to  influence  the  answer,  he  is  using  a  "leading"  question. 

The  difference  between  the  two  types  of  "leading"  questions  is  that  in 
the  first  case  the  evaluator  has  factual  information  to  support  the  question. 
In  the  second  ^ase  he  lacks  the  information,  but  "leads"  the  graduate  in  the 
direction  of  the  interview  results  he  wants. 

Trie  second  element  in  tne  "leading"  question  is  based  on  the  fact  that 
there  are  areas  of  commonality  among  all  people.  Because  of  this  it  is 
almost  imoossihie  to  ask  a  question  that  is  not  based  on  some  type  assumption. 
If  the  assumption  is  based  on  valid  information  in  the  possession  of  the 
evaluator,  the  graduate's  answer  will  be  informative.  If  the  evaluator  does 
not  have  specific  information  upon  which  to  base  his  assumption,  this  type 
"leading"  question  may  produce  incomplete  or  invalid  information. 

ft  is  difficult  to  determine  the  detrimental  effects  of  the  "leading" 
question  on  the  results  of  tne  interview.  However,  use  of  the  "leading" 
question  does  make  it  easier  for  those  graduates  who  lack  the  knowledge,  or 
who  do  not  understand  what  tne  evaluator  wants,  to  give  an  answer  they 
believe  the  evaluator  wants  to  hear.  Probably,  these  "agreeable"  answers  are 
most  often  given  by  those  having  a  low  level  of  understanding  than  by  those 
having  a  high  level  of  understanding. 

Just  a  little  more  on  tie  subject  of  questions.  Questions  may  be  asked 
for  an  objective,  subjective,  or  indeterminate  response  from  the  graduate. 

The  objective  question  is  concerned  with  the  characteristics  of  people, 
places,  objects,  or  happenir  is,  that  can  be  seen.  These  are  the  type  question 
used  when  a  schedule  of  questions  is  prepared.  These  are  fact-finding 
questions. 

The  subjective  question  deals  with  opinions  or  feelings,  the  unseen 
factors,  these  questions  ara  used  for  the  third  type  of  interview.  Every 
answer  must  be  analyzed  to  determine  its  reflection  of  the  attitude  of  the 
qradua te. 


Indeterminate  questions  deal  neither  with  characteristics  or  opinions, 
fuf  are  often  descriptive  in  tone.  These  too,  are  used  in  the  third  type  of 
interview.  Like  the  subjective  question  answers,  they  must  be  analyzed. 
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You  have  asked  a  lot  of  questions  and  heard  many  answers,  how  do  you 
know  if  the  graduate  has  told  the  truth,  if  his  answers  are  valid?  The 
evaluator  is  responsible  for  determining  the  validity  of  what  the  graduate 
tells  him  and  he  has  available  to  him  three  methods  for  estimating  the 
validity. 

If  the  evaluator  has  properly  prepared  himself  for  the  interview  he  has 
considerable  amounts  of  information  at  his  disposal.  By  careful  analysis  of 
what  he  is  told  and  by  comparing  it  with  what  he  knows  of  the  work  area,  he 
is  in  a  position  to  make  reasonably  accurate  judgments  of  the  validity  of  the 
information  he  has  received. 

Another  method  of  estimating  the  validity  is  the  observation  and  evalua¬ 
tion  of  the  style  and  manner  in  which  the  graduates  have  responded.  Close 
observation  during  the  interview  wi 1 1  often  reveal  clues  indicative  of  the 
sincerity  of  the  graduate  and  thus  key  the  authenticity  of  his  answers.  If 
the  graduate  shows  any  doubt,  hesitancy,  prejudice,  or  uncertainty,  the 
information  should  be  accepted  with  caution. 

Finally,  the  evaluator  should  be  able  to  make  some  judgments  based  on 
what  ne  knows  about  the  graduate.  An  insight  into  the  graduate's  knowledge, 
experience,  or  background  will  yield  clues  to  the  worth  of  his  answers. 

The  entire  success  of  the  interview  depends  upon  the  degree  of  communi¬ 
cation  established  between  the  evaluator  and  the  graduate  and  the  ability  of 
the  evaluator  to  listen  intelligently.  The  development  and  format  of  the 
interview  must  have  as  its  goal  the  complete  understanding  and  cooperation  of 
the  graduate.  The  evaluator  must  earn  the  respect  and  the  confidence  of  the 
graduate.  Any  action,  any  words  which  weaken  the  respect  and  confidence  of 
the  graduate  in  the  evaluator  will  decrease  the  effectiveness  of  the  interview. 

It  is  highly  possible  that  in  certain  situations  the  evaluator  will  be 
confronted  with  some  unanticipated  responses.  If  he  is  alert,  these  unanti¬ 
cipated  responses  may  be  highly  productive.  An  unanticipated  response  may 
reveal  previously  unsuspected  important  factors,  it  may  open  the  door  to  new 
avenues  of  questioning,  or  it  may  provide  an  explanation  of  some  hitherto 
unknown  fact. 

The  conduct  of  the  interview  is  entirely  in  the  hands  of  the  evaluator. 

In  today's  vernacular,  he  must  not  lose  his  "cool",  if  he  does,  he  also  loses 
the  interview.  At  all  times  and  at  all  costs,  the  evaluator  must  be  in  full 
control  of  the  interview.  Any  lack  of  organization  of  the  interview  will 
also  have  an  adverse  effect  on  the  interview  result.  It  is  essential  that 
the  interview  continue  in  the  direction  of  its  established  goals.  At  no  time 
and  under  no  circumstances  must  the  interview  be  allowed  to  degenerate  into 
an  argument. 

The  graduate  and  his  supervisor,  either  or  both  of  whom  may  be  inter¬ 
viewed,  must  be  considered  as  having  a  defensive  attitude.  Neither  will 
volunteer  any  significant  quantity  cf  information.  Each,  in  his  own  way, 
will  defend  and  magnify  the  importance  of  their  job  to  the  Air  Force  mission. 

No  matter  how  insignificant  the  job,  it  is  theirs  and  admitted  or  not,  they 
are  proud  of  what  they  do.  Destroy  the  importance  of  their  job  and  you 
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destroy  them. 

It  is  important  that  the  evaluator  show  sympathetic  interest  in  the 
problems  of  the  graduate.  Don't  interrupt  his  story  to  tell  yours.  Never 
show  boredom,  even  if  you  have  heard  the  same  story  twenty  times  during  the 
same  day. 

The  evaluator  must  never  allow  controversial  issues  to  enter  into  the 
conversation.  If  they  are  introduced,  listen,  but  offer  no  comment  and  at 
the  first  opportunity,  change  the  subject.  Never  become  a  victim  of  rumor 
mongering.  Listen  carefully,  but  do  not  repeat  what  has  been  said.  Squelch 
those  rumors  you  know  to  be  untrue. 

The  evaluator  has  no  authority  to  promise  changes  effecting  the  graduate's 
assignment  or  position.  To  do  so  must  lead  to  embarrassment  for  the  Air 
Traininq  Command,  the  Center,  and  himself.  The  interview  is  a  fact  finding 
vehicle  for  evaluating  the  quality  of  the  training  received  by  the  graduate, 
it  is  not  a  grievance  interview.  Stay  out  of  the  personnel  and  the  personal 
business. 

Do  not  criticize  any  action  affecting  the  graduate  or  the  operations  of 
any  Air  Force  function.  It  is  probable  that  not  all  the  facts  have  been  made 
known  to  you.  Get  what  you  believe  to  be  facts,  verify  that  they  are  facts. 
Analyze  their  probable  impact  on  the  performance  of  the  graduate,  or  upon  the 
conduct  of  the  traininq.  Report  the  findings,  include  any  supportable 
recommendations . 

When  the  interview  is  completed  thank  the  participants  for  their  help. 
Leave  them  with  a  friendly  feeling  toward  training.  It  is  highly  probable 
they  will  again  be  called  upon  to  assist  in  evaluating  graduates.  Without 
their  full  cooperation,  there  would  have  been  no  interview,  no  exchange  of 
information,  and  no  evaluation  of  the  graduate's  performance  or  of  the  course 
from  which  he  graduated. 
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ATTACHMENT  4 

AN  EXPERIMENTAL  AGENDA  FOR  DETERMINING  HOW  TO 
SOLVE  TRAINING  PROBLEMS 

This  attachment  contains  an  experimental  agenda  for  processing  the 
feedback  information  obtained  in  che  interviews  to  a  logical  conclusion--a 
recommendation  for  change.  This  agenda  is  offered  for  use  with  the  job 
tasks  listed  in  the  survey  for  which  the  data  suggested  a  possible  training 
problem.  The  process  involves  the  following  10  steps: 

1.  Define  the  training  problem  as  suggested  by  obtained  feedback. 

2.  Review  of  the  background  (i.e.,  previous  attempts  to  deal  with  the 
problem) . 

3.  Review  of  related  information  from  internal  evaluation  process. 

4.  Identify  related  learning  objectives  in  the  current  curriculum. 

5.  Review  of  recent  curriculum  development  with  regard  to  the  relevant 
learning  objectives. 

6.  Explore  resources  to  remediate  problem. 

7.  Identify  related  training  problems  for  which  the  solutions 
may  also  be  related. 

3.  Rough  outline  of  possible  recommendations  and/or  entrance  point  in 
ISD  process  for  further  systematic  development. 

9.  Determine  approval  authority  for  outlined  recommendation  and 
list  pertinent  instructions/references. 

10.  Recommendation( s ) . 
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Definition  of  training  problem: 


Background:  (Historical  attempts  to  deal  with  this  problem  or  related 
problems,  except  for  recent  developments  in  the  last  year.) 


Relevant  data  from  internal  evaluation  process: 
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4.  Pertinent  learning  objectives  from  the  current  curriculum: 

_ i  _ _  _ _ 9  _ _ _ _ _ _ 9 _ _  _ _ 5  _ 

b.  Recent  curriculum  development  in  these  areas: 

Recent  cr  anticipated  change  in  time  allotted  to 
teaching  this  or  closely  related  task:  -  hours:  + 


Recent  or  anticipated  change  in  teaching  technique: 


Recent  or  anticipated  change  in  resources  (manpower/ 
material)  that  nay  affect  training  in  this  area: 


b.  Resources  required  to  remediate  problem: 

Additional  space  for  instruction 


More  effective  training  ai ds/deviccs 
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Additional  instructor  manpower 


Additional  time  for  instruction  added  to  the  curriculum 


Recent  and/or  anticipated  changes  to  curriculum  will  remediate 
the  problem  (time,  teaching  technique  or  resources) 


Recent  and/or  anticipated  changes  to  curriculum  will  increase 
the  problem  _ 


Creative  brainstorming  and/or  additional  research  into  more 
effective  teaching  strategies  (problem  is  historical  in  nature 
requiring  entirely  novel  approach  or  additional  data) 
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7.  Related  tasks  from  the  survey  for  which  the  solution  to  this  problem  may  be 

related:  _ ,  _ ,  _ ,  _ . 

8.  Rough  outline  of  possible  recommendations:  (Address  issues  of  cost  and 
anticipated  benefit)  and/or  entrance  point  in  ISD  process  (see  NAVEDTRA  110)  for 
further  systematic  development: 


9.  Approval  authority  for  this  type  of  recommendation: 


Pertinent  Instructions/References : 
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